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ABSTRACT 

This report describes an extensive field test of the 
Bilingual Education Evaluation System (BEES) used to evaluate local 
level bilingual education projects. Because such projects will 
usually not be able to implement a traditional true or 
quasi-experimental design, BEES employs a jap-reduction" evaluation 
design that is easily implemented, satisfies regulatory requirements, 
and yields an index relating to the growth made by project 
participants without requiring a comparison group of students. This 
evaluation method was tested at nine school districts in six states. 
This report describes this field test and the training and assistance 
program offered to participating educators. The report also 
summarizes the participants' comments and suggestions about how the 
evaluation system might be improved. In particular, it is suggested 
that the written materials be improved and that user-friendly 
software be provided to assist in the database management tasks and 
to perform the gap-reduction computations. Extensive documentation of 
the field test is appended including survey instruments and sample 
forms. (JL) 
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I. INTRODUCTION 



The Evaluation of Bilingual Education Programs for Language-Minority, 
Limited- English Proficient Students, currently being conducted by RMC 
Research Corp, under contract to the U.S. Office of Education, is intended 
to improve local evaluation practices with the dual goal of enhancing 
local utility of evaluation information and providing a data base that 
will be useful for broader purposes. As a first step in the study, an 
extensive review of the literature was conducted to determine the state of 
the art in bilingual education evaluation in the United States and to 
develop recommendations for a bilingual evaluation system. The second 
step was the development of a methodologically sound, standardized evalua- 
tion system and its documentation in a two-voltime Users' Guide . The Bil- 
ingual Education Evaluation System (BEES) was designed for use in evaluat- 
ing local level bilingual education projects. 

Development of the BEES was guided by four primary design objectives: 

9 The system should reflect the knowledge gathered out of previous 
work in bilingual education evaluation. 

• It should provide g^^aance on how to avert salient threats to the 
validity of evaluaticr. findings. 

• It should be useful at the local level for purposes of project 
improvement . 

• It should be totally responsive to the current federal legislation 
and regulations governing the evaluation of Title VII projects. 

In developing the BEES we assumed that most projects will not be able 
to implement a traditional true or quasi-experimental design for reasons 
that are well documented throughout the literature. For this reason, we 
formulated a "gap -reduction" design that i.^ easy to implement, satisfies 
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the regtilatlons' requirements, and does not require a comparison group 
made up of students similar to those served by the project. 

« 

The gap-reduction design yields an index relating the growth made by 
project participants to that made by a non-project comparison group rather 
than a quantitative estimate of the project's impact. The evalxiator must 
rely on a wide array of information and findings about both process and 
outcome to form judgements about project effectiveness. The system empha- 
sizes quality control and recognition of unavoidable problems as a method 
of avoiding erroneous conclusions. 

The third step of the study was to field test the system in nine 
school districts across the country. The purposes of the field test are 
to answer the following c^nestions about the BEES: 



e To what extent has the system aided the sites in defining and 
objectively measuring the impact of the bilingual program on its 
students? 

• To what extent can the system serve to improve local evaluation 
practices? 

• To what extent can the system generate data suitable for use in 
local decision making? 

• How adequately can the evaluation designs be integrated with a 
district's regular instructional and testing programs? 

• How suitable are data generated by the system for synthesis across 
Title VII projects? 

© What are the costs incurred in implementing the system's evalua- 
tion designs (costs associated specifically with the project site, 
start-up costs, and ongoing costs)? 
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• To what extent would a district require technical assistance in 
implementing the system? 

While participation in the field test has represented some burden to 
the sites, this has been offset by training and technical assistance pro- 
vided by RMC staf f • These 'activities will be described in the next chap- 
ter of the report. 

Oxir goal in selecting field test sites was to represent the broadest 
possible range of bilingual education projects in terms of geography, 
minority languages, and program design* Site selection criteria are 
described in Appendix A* We decided that because California, Texas, and 
New York have the largest bilingual populations, they should each have two 
sites* Arizona, Florida, and Oregon were selected to participate with one 
site each. 

sea's were asked to nominate sites for inclxision in tha study, and 
sites were then asked to participate. Appendix B is the invitation letter 
sent to sites* Two initially selected sites dropped out and were 
replaced* We were not able to locate any late-exit or inmiersion projects 
willing to serve as field test sites* Each field test site is described 
briefly below and stimmarized in Table 1* 

Tucson Unified S,D,, Arizona 

The Tucson Unified School District serves 54,000 students, of whom 
26% are language minority and 22% are Spanish speaking* language -minority 
students are over- represented in special education (13*5% compared to 8% 
district-wide) and under- represented in gifted classrooms (*3% compared to 
4% district-wide) * Even lower percentages of LEP students are in gifted 
classrooms • 

Bilingual special education classes are funded by the district* The 
Title VII project supplements and expands these programi^ in grades K-12 



and has encbled the establishment of a bilingual gifted class for gifted 
LEP students. The project offers transitional bilingual education to 
identified students. When the students become English proficient, they 
have the option of transferring to comparable all-English classrooms or of 
remaining in bilingual education. Approximately 228 studciiis are served, 
of whom 85% are low incoice. 

San Jose Unified School District. CA 

San Jose Unified School District, located south of San Francisco, 
serves approximately 217,704 studerts (of whom CSX are from low- income 
families and 12% are LEP) representing at least 36 different languages. 
One of the fastest growing language groups in the district is Indochinese. 
The Indochinese LEP population grew from 455 students in 1981 to 766 in 
1986. 

The Title VII project is located in five schools, elementary through 
high .school, which were identified as having a significant influx of Indo- 
Chinese students 'and having demonstrated a long-term commitment to meeting 
the. needs of Indochinese LEP students. The project includes ESL itistruc- 
tion for all Indochinese students, LI instruction for Vietnamese -speaking 
students and for other language groups where possible , staff and parent 
training, and curriculttm development. 

Vdllev C enter Union School District. CA 

The Valley Center Union School District, in a rural part of San Diego 
County, serves over 1,300 students at grades K-8. The population includes 
commuters, ranchers, citrus farmers, and ranch- and farmworkers. Approxi- 
mately 12% of the district's students are Spanish -dominant LE? students. 
The district has provided ESL and Spanish reading instruction to these 
students, who by grade 8 lag approximately five years below the grade 
level of their English-speaking peers while being chronologically two 
years older. 
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The project is a transitional bilingual education project emphasizing 
concept and skill development in Spanij^h with later transfer to English • 
The project will serve 162 LEP students at grades K-8, 74% of whom are 
from low- income families • The project uses as basal readers language 
texts centered around each child's unique experiences ♦ The project also 
includes Spanish language development for English- only students ♦ 

Comniunitv School District 4t5. New York Cttv 

Coiranunity School District #5, in economically-deprived Karlem, has 
approximately 11,911 students enrolled in its public schools and 3,162 in 
non-public schools* In the puMic schools, there are 836 LEP Hispanic 
students • The families of many of these children have recently immigrated 
from rural areas offering few or no school services* Over half the par- 
ents have less than an eighth- grade education. 

The Title VII project is located in P.S. 161, an elementary school 
with a large concentration (over BOX) of Hispanic students which continu- 
ally receives new arrivals from Hispanic countries • This school also 
houses a district bilingual program and a Chapter 1 program. The Title 
VII project will serve new arrivals in grades K to 3, new incoming kinder- 
garten and first graders, and second and third graders who score at or 
below the 20th percentile on the lAB, a total of about 258 students • The 
project is a transitional bilingual education project with an emphasis on 
developing students' skills in both Spanish and English* 

School Board of Polk County. FL 

Polk County is a 58,000 student school district located in central 
Florida. Citrus cultivation, one of the county's main industries, 
attracts a large population of Spanish- speaking migrant farmworkers to the 
county. In 1981-82, Polk County schools served 6,683 children of migrant 
workers. Four percent of the district's minority- language students are 
referred each year for serious intellectual, academic, social, and/or emo- 
tional problems. 
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A district -funded bilingual education program has been in place since 
1979-80- The Title VII transitional bilingual education project has been 
implemented to bring additional services to district schools having the 
highest concentrations of LEP Hispanic students. Goals of the project are 
to meet the academic needs of these LEP students to enable them to achieve 
at the rate and level of their non-LEP peers and to provide students and 
their families with needed services through employment of a Hispanic 
social worker. The project serves approximately 380 students in grades 
K-8. 

Tvler Independent School Di strict. TX 

Tyler Independent School District in Northeast Texas is a district of 
16,500 students, of whom 3% are Spanish -dominant and LEP. The number of 
LEP students has been increasing dramatically. These students on the 
average are achieving from one to three years below grade level, and most 
are from families of very low socio-economic status. The district suffers 
from a shortage of teachers with bilingual education credontials. 

The Title VII transitional bilingual education program, which serves 
approximately 300 students, operates out of two elementary school build- 
•ings in attendance areas having the largest numbers of LEP students. Stu- 
dents in other attendance areas are bused to one' of these buildings. 
Staff development,, in conjunction with a nearby university, is an impor- 
tant part of the project, so that local Spanish -speaking teachers and pro- 
spective teachers can obtain .appropriate credentials. 

Valley View Indepe ndent School District. TX 

Valley View Independent School District, located in a rural part of 
Texas along the Rio Grande, serves 746 students. Of these 98% come« from 
low-income families and 88Z are LEP. Ninety-f;ive percent of the dis- 
trict's population is of Mexican descent, ^x:d 65% are migrant field work- 
ers, many of whom are recent immigrants. LEP third and fifth graders in 
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the district achieve less than SOX mastery of objectives on the Texas 
Assessment of Basic Skills. 

The Title VII project is located in the district's elementary school 
site. It serves approximately 550 students in grades K-5 with a program 
emphasizing concurrent development of English and Spanish reading skills. 
The project has a aaterials development component involving the develop- 
ment of computer software for the reading instruction. Forty parents of 
children with severely limited English proficiency will be recruited for a 
special parent involvement program emphasizing literacy and transfer of 
skills to English. 

Salem-Keizer Public S chools. OR 

In the fall of 1985, Salem-Keizer Public Schools began serving dis- 
trict five-year olds for the first time. Half of district students and 
over 80% of district LEP students had in the past been found tmable to 
enter first grade at age six. Because the district offers bilingual pro- 
grams at first and successive grades, there was a need to offer a bilin- 
gual program for five-year old LEP studen^s who would then continue with a 
bilingual program in the first grade. 

Four elementary schools with the highest concentrations of LEP stu- 
dents in the district were selected for the program, which includes stu- 
dents whose primary languages are English, Vietnamese, Khmer, and Spanish. 
The goal of the program is to enable studerts to be promoted to first 
grade and through successive grades. The program includes a home- teaching 
component for parents, and serves approximately 100 students. 

CoTTpunitv School , District V/10. New yr^x ^ 

Community School District #10 is located in the northwestern portion 
of the Bronx. It has a highly diverse, multi-ethnic student population of 
about 35,000 students in grades K-9. Approximately 20Z are LEP students 
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of Hispanic or Asian backgrounds • Four intermediate and junior high 
schools in the district have been identified, as most in need of assis- 
tance, based on mobility and truancy rates and percentages of lip stu- 
dents* Over half of these LEP students are Hispanic. 

The Title VII project provides supplementary services and opportiini- 
ties to those Spanish-dominant students judged most in need of assistance. 
Services include ESL; self -awareness; decision making; career, economic, 
and educational awareness , and employability skills . Students also 
receive assistance from a bilingual education assistant and a family 
worker. Approximately 300 students are served. 

Contents of the Report 

This report describes the field test of the bilingual evaluation sys- 
tem in these nine districts. Chapter 2 discusses training and technical 
assistance offered by "HC to the field test sites to assist their imple- 
mentation of the evaluation system. Chapter 3 summarizes comments and 
suggestions for changes for each of the 10 chapters in Voltune 1 of the 
Users ^ Guide gathered during the field test, and Chapter 4 summarizes the 
lessons learned from the Field Teat. 
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Table 1 



Np* Grade Predom* 

Current ^ Students Levels Language Program 
Site Project Year Served Served Groups Type 



Tucson Unified 3 
S*D* , Arizona 



School Board of 1 
Polk County, FL 



Community S.D. 1 
#10, New York 

San Jose Unified 3 
School District 



Conimunity School 1 
District #5, New 
York 

Valley Center 3 
Union S*D*, CA 



Salem-Keizer 2 
Public Schools, 
OR 

Tyler Independent 1 
S.D., Tyler, TX 



Valley View 3 
Indep enden t S . D . , 
TX 



228 



380 



K-12 Spanish 



300 
360 

258 

162 

99 

300 

550 



K.8 



6-9 



K-12 



K-3 



K-8 



K-S 



K.5 



Spanish 



Spanish 



Vietnam- 
ese, other 
Asian 

Spanish 



TBE-special 
ed* and gifted 
classes* 

T5E and 
support 
services for 
migrants • 

ESL, career 
education 

ESL and TBE 



Span: ;h 



TBE, enphasis 
on thinking 
skills 

TBE, home 

teaching 

component 



age 5 Multiple TBE 



Spanish 



Spanish 



TBE, emphasis 
on staff 
development 

TBE, materials 
development, 
parent educa- 
tion 



II. SRA TRAINING AND TECHNICAL ASSISTANCE ACTIVITIES 
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• Two workshops were scheduled during the field testing, one in Octo- 
ber. 1986 and one in June. 1987. The purpose of the first workshop as to 
train the field test participants on the bilingual Education- Evaluation 
System (BEES) . The June workshop or exit conference was designed to pro- 
vide an opportunity for these participants to share their experience in 
trying out the BEES and to provide input for further revision of the 
JJsers' Gy^^lR . Agendas and correspondence relating to these workshops can 
be found in Appendix C. 

The October workshop was held from October 7-9 in San Francisco. 
During the first morning. Lam presented an overview of the BEES and dis- 
cussed evaluation planning and focusing in the BEES. In the afternoon. 
Lam discussed process evaluation. Workshop participants took a pretest at 
the beginning of the morning session, and a posttest at the end of .the day 
so that they could measure how much they had learned. 

On the second day. Tallmadge discussed outcome evaluation, including 
measurement error and the regression effect, selecting achievement tests, 
other indicators of achievement, test administration and scoring, calcu-' 
lating growth indices, and quasi-experimental designs. Workshop partici- 
pants took a pretest designed to sensitize them to the issues that would 
be presented that day. There was no posttest. however. 

On the third day. Tallmadge and Lam discussed data processing and 
analysis, integration and interpretation of results, and evaluation 
reporting. At the end of these presentations, the workshop turned to an . 
open discussion of the Users' Guide and the field test. 

After this meeting, follow-up letters were sent to participants to 
clarify their roles in field testing the Users' Gt HdP. This letter high- 
lighted key activities desired of field test site staff, asking them spe- 
cifically to implement Users' Guide recommendations or document why the 
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reconmendations were not practical in their districts • A similar letter 
was sent to the sites in April in preparation for the June, 1987 workshop. 

The June workshop was held on June 17-19 in Washington, D*C* This 
workshop was attended by foxir representatives from Tyler, Texas; two rep- 
resentatives each from Communitrjr School District #5, New York; Communitgr 
School District #10, New York; Valley Center, California; San Jose, Cali- 
fomia; and Valley View, Texas; and three representatives each from Tuc- 
son, Arizona; Salem-Keizer, Oregon; and Polk County, Florida. Additional 
participants during all or part of the workshop were James English and 
Keith Baker from the Education Department, Office of Planning, Budget, and 
Evaluation; Pat Johanson, from the Education Department, Office of Bilin- 
gual Education and Minority Language Affairs; and various staff from the 
Education Assistance Center - East. Tallmadge and Lam represented the 
project staff • The first day of the workshop was devoted to presentations 
by each of the field- test sites. A staff member from each site presented 
an overview of the site's Title VII project,, the activities undertaken to 
implement each Users' Guide recommendation, and a description of the 
results of the implementation, including any problems • 

The second day and part of the third morning was devoted to a chap- 
ter-by-chapter review of the Users' Guide by field site staff. Site 
project staff divided into four subgroups (New York districts; Salem and 
Tucson; Valley Center, San Jose, and Valley View; and Tyler and Polk 
County). Tallmadge met separately with attending project evaluators to 
review Chapters 6 (Implementing the Gap-Reduction^ Design) and 8 (Process- 
ing and Analyzing Data). Review of the chapters by project staff took 
longer than expected and, as a result, "only the first seven chapters were 
covered during the workshop (Chapter 8 was reviewed by evaluators only) . 

Two questionnaires were also administered during the workshop (see 
Appendix D) . One questionnaire was designed to ascertain the partici- 
pants' perception of the importance of the evaluation data reconmiended for 
collection on students, instruction, and staff for estimating program 




effectiveness and for improving project performance ♦ This questionnaire 
was a follow-up of another questionnaire sent to the participants prior to 
the June workshop ♦ In this earlier questionnaire, each pilot site was 
asked to rate the availability and accessibility of the data included in 
the later questionnaire. The second questionnaire that was administered 
during the workshop was designed to obtain input from the nine sites re- 
garding the importance and tiseftdness of the community characteristics, 
district demographics, and school characteristics data recommended by the 
BEES. The restilts of these three qtiestionnaires are discussed in Chapter 

Part of the third morning was used to summarize findings from the 
first two days. The findings from this meeting and from the field testing 
with regard to tkie implementation of the Users ^ Guide are presented in 
C3iapter 4. 

In accordance with the Phase III modified contract, two site visits 
were made to each site by either Law or Tallmadge. On the average, each 
visit lasted two days. Table 2 lists the dates for all these site visits. 
As Table 2 shows, site visits were scheduled at varying intervals, depend- 
ing upon the availability of RHC staff and the wishes of site personnel. 
Lam visited Tucson, San Jose, Valley Center, New York Districts #5 and 10, 
and Salem. Tallmadge visited Polk County, Florida, and Tyler 'and Valley 
View, Texas. 

The conduct of each site visit, was qtdte similar. Each visit 
incltided a meeting with the site's bilingual project staff, individually 
or in a group, to discuss the site's program and evaluation designs as 
they related to the recommendations embedded in the Guide. For example, 
project staff might have been concerned that the test used by the project 
to measure academic achievement was too difficult for the target popula- 
tion. Since the Guide recommonds the use of functional-level testing, 
this alternative testing procedure would have been proposed by RHC staff 
and discussed with the group. 




In addition, for some sites, we conducted classroom observation with 
the project site staff aither to determine the project's critical instruc- 
tional features or to try out a newly developed classroom observation 
checklist. The sites for which classroom visitation was conducted include 
San Jose, Valley Center, Salem, and Tucson. 

In addition to site visits and these workshops, we offered sites 
technical assistance through frequent telephone conversations and periodic 
correspondence . 

Since outcome data were not available, the technical assistance pro- 
vided thus far has focused primarily on evaluability assurance, evaltiatioa 
planning, process documentation, data collection and management, and 
assessment instrument review. It is anticipated that after the June meet- 
ing, limited technical assistance will be provided on data analysis and 
reporting. In Table 3, different categories of technical assistance acti- 
vities and the sites receiving each type are summarized. 




13 



17 



Table 2 - Site Visitation Dates 



Site 

Tucson Unified 
School District, AZ 



1st Visit 



2nd Visit 
5/11-13/87 



San Jose Unified 
School District, CA 

Valley Center Union 
School District, CA 



10/24/86 
1/22/87 

2/16-17/87 



5/7/87 
6/9/87 

4/27-28/87 



Polk Covinty School 
District, FL 



3/17-18/87 



5/12-13/87 



Connnunity School 
District //5, NY 



3/18/87 
3/20/87 



5/27-28/87 



Connnunity School 
District #10, Bronx 



3/17/87 
3/20/87 



5/26-27/87. 



Salem-Keizer Public 
Schools, OR 



2/23-24/87 



5/4-5/87 



Tyler Independent School 3/10-11/87 
District, TX 



5/14-15/87 



Valley View Independent 4/7-8/87 
School District, TX 



6/5/87 
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Table 3. Areas of Technical Assistance by Site 



Site 



Areas of Technical 
Assistance 



Tucson 


Salem 


Polk 
County 


Tyler 


Valley 
View 


Valley 
Center 


San 
Jose 


NY 

m 


m 

#5 


X 


X 








X 


Y 


Y 

XL. 


V 
A 


X 


X 








X 


X 


X 


X 


X 


X 


X 


X 




X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 




X 


X 


X 


X 




X 


X . 


X 


X 


X 


X 


X 




X 


X 


X 










X 




X • 








X 


X 


X 






• 
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o Revise project 
objectives 

0 Elaborate on 
project design 

o Develop classroom 
observation checklist 

0 Review data to be 
collected 

o Manage data 

o Review tests and 
testing practices 

0 Review affective 
outcome instruments 
and methods of effec- 
tive assessment 

o Data analysis 
(including gap- 
reduction analysis) 

•o Reporting 
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In the remainder of the chapter, we describe, site-by-site, our con- 
sultation with the nine field-test sites. We focus on the unique evalua- 
tion problems encountered by each site, the recommendations provided by us 
in dealing with these problems based on the Guide , the problems in imple- 
menting these recommended practices, and the sites' needs for additional 
technical assistance « 

Tucson Unified S>D>> Arizona 

The objectives stated in the Tucson Unified School District Title VII 
project's proposal are all process objectives that deal with identifying 
and assessing LEP gifted students, establishing a bilingual gifted class, 
expanding bilingual special education, training staff and involving par- 
ents. Expected student outcomes are only briefly mentioned. 

As stated in the proposal, students' literary skills in Spanish and 
English, and their math skills are expected to be eiihanced as a conse- 
quence of their participation in the project. Spanish reading skills 
and math skills are measured by the Bateria Woodcock (K-12) , La Prueba 
(K-8), and the Brigance Diagnostic Assessment of Basic Skills (K-8) tests. 
Skills in English reading are measured by the Woodcock and the Brigance 
tests after the students are exited from the bilingual to the regular spe- 
cial-education classes* 

The La Prueba test is a norm- referenced test and is administered to 
all district LEP students. The Woodcock test is a norm- referenced test 
and the Brigance test is a criterion-refer^^nced test* Both tests are xised 
for identifying students in need of special educational service and for 
diagnosing the students' strengths and weaknesses. This diagnosis forms 
the basis for developing an Individualized Education Plan (lEP) for each 
student. This individualized instructional approach has made it difficult 
to establish a general criterion of success (as measured by test scores) 
for all students, since the expected growth rate varies among students. 
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One reconnaenda'cion for assessing tlie project's effect on student out-' 
comes is to examine each student's performance individually. One way to 
accomplish this is to calculate the percentage of incorrect responses on 
the Brigance pretest (each student is administered a different set of 
items depending on his functional level) that are correctly answered dur- 
ing posttest. Since each student has his own set of objectives, another 
way to assess project impact is to compute the percencage of objectives 
completed, partially completed, and not completed for each student. 
Criteria for success for all outcou^e objectives based on these statistics, 
and project performance can be ev&l\iated based on the extent to which 
these criteria are met. Undoubtedly, stakeholder bias could be severe 
with this approach. 

The La Prueba test results will also be useful for comparing project 
and district LEP students. Since the test is designed for regular stu- 
dents, and not for students with physical, language and/or learning dis- 
abilities, functional- level testing should be used and some modification 
of the test instructions should be made. The project has done both. The 
test modifications include allowing students to mark their answers on the 
test booklet, extending the time limits, and giving 3iore elaborate 
instructions. Since the test can only be xxsed at grades K-8, the Woodcock 
test will have to be administered to students in grades 9 through 12. 

Due to its individualized instructional approach. It is also diffi- 
cult for the project to prescribe the specific teaching approach that all 
project teaching staff should follow. During the two site visits, efforts 
were made by Lam to assist the pvojcct staff in gerxerating a list of crit- 
ical instructional features. After tituch discussion and some classroom 
observation, it was decided that the observational scheme should incorpo- 
rate the variety of the lEPs. Instead of specifying desired teaching 
behaviors, three general questions were developed to determine levels of 
program implementation through classroom observation. These questions 
are: 
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• Are the.IEP objectives appropriate for the student? 

• Is this instructional approach appropriate to the lEP objectives? 

• Is the language appropriate for the student as determined by the 
language of instruction prescribed in the lEP? 

Specific indicators to guide responses to these three questions will 
be developed after some initial classroom observations ♦ In addition, a 
teacher survey was developed to assess primarily how teachers use the two 
languages to conduct instruction (see Appendix E). 

Technical assistance wafe also provitied in data management. In part 
because of the reluctance of the project teachers to test students and 
record test results systematically, test scores for the first two years of 
the project are so incomplete that they cannot be used for program evalua- 
tion* There was no system for filing evaluation data once they were col- 
lected* Since the district has decided to participate in the field test, 
.efforts have been made to centralize the data and to ensure complete test- 
ing* Teachers have received in-service training on better data recording • 
We will continue working with project staff to develop a data management 
system* 

Salem-Kelzer Public Schools 

As stated in the district's proposal, the project has foa^ goals, 
related to establishing and implementing an instructional program for 
5 -year old LEP students, involving parents, and training project staff* 
The two student outcome objectives specify expected achievement in English 
competence as measured by the LAS, and the extent to .which the project 
students will meet the requirements of the district's five-year-old pro- 
gram as measured by the Salem Skills Checklist developed by the district* 

One of the major difficulties in conducting evaluation for the Salem 
project is testing 5 -year olds* Standardized testing instruments are not 
readily available, which makes implementation of the gap-reduction design 



difficult because of the lack of normative data* It is not clear if the 
IAS is appropriate for measuring achievement in English language profi- 
ciency for this age group* The problem of curriculum- test match may still 
exist for this grade levels Although it is subjective, the Salem Skills 
Checklist, which is designed to measure skill levels in physical coordina- 
tion, perceptual memory, language, and mathematics appears to be a viable 
instrument for measuring student achievement • Since pretest results from 
the checklist are used to prescribe instructional activities and the stu- 
dents are poJttested on those areas in which, they receive instruction, 
there is a perfect curriculum- test match* Again, using the checklist 
makes it difficult to implement the gap-reduction design. 

There is a third implicit student outcome objective which relates to 
increasing student understanding of language and cultural heritage. The 
objective is very difficult, if not impossible to measure directly. Other 
indirect measures of the attainment of the objective, such as frequency of 
cu;itural activities and student behavior and participation in these acti- 
vities, were proposed. 

Procedures to document the level of instructional sejcvice delivery 
consumed a great deal of effort. During the initial meeting with the pro- 
- ject staff, we brainstormed on the critical instructional features to 
determine ways to document the extent to which they are carried out in the 
classes. The first classroom observation instrument developed was a 
behavioral checklist that requires tallying occurrences of specified 
behaviors. When Lam and the project staff tried it out in the classroom, 
they found that it was not feasible to record the many instructional and 
student activities that are going on in the class. It was decided that 
the delayed- report approach is more appropriate. 

After a few revisions, an observation form was developed that the 
project staff feel is useful and appropriate for documenting the project's 
instructional service delivery (see Appendix F) . The form can be used to 
record activities for several small groups or for a single group across 




different time segments. The observation system requires the obsexrver to 
write down for each gvoup or each time segment, the instructional metho- 
dology employed and the language used for instruction and their appro- 
priateness for the students involved. Although the form is easy to use, 
the observers must be very knowledgeable about bilingual and ESL pedagogi- 
cal methods. This is the tradeoff between structured versus non- 
structured observation that needs to be addressed in the Guide . 

Another area in which the project needs technical assistance is in 
data management. Evaluation, data are being collected from different 
sources : forms, student records, and district evaluation offices' computer 
files. The project plans to utilize its IBM computer to manage a?! the 
data. Continued assistance to integrate all these data into a computer- 
ized data base will be needed in the coming academic year. 

Polk County. Florida 

Technical assistance/field test visits were made to the Polk County 
school district on March 17 and 18 and May 12 find 13, 1987.- 

Th^-Hayph vi?it> During the March meeting, activity was focused on a 
page-by-page review of the Users' Guide with the project .director and the 
district evaluator. Those persons had studied the Guide and had assessed 
the feasibility of implementing its recommendations. Many of the recom- 
mended practices and procedures were already in place. No recommendations 
that were not already in place had actually been implemented in the dis- 
trict, but many were scheduled for implementation next year. 

In some cases. Polk Coxinty felt that their current practices had at 
least some features that were^ superior to the Users' Guide recommend- 
ations* In other instances, they preferred our recommendations to their 
current practice. In a few instances, the site persons identified con- 
flicts between official state or district policies and our recommend- 
ations. All of these specific comments are described below. 
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The only general comment that site personnel made was that the Users* 
guide was calling for the collection and sununarization of substantially 
more information than they felt was required by the regulations and more 
than they or anyone else could use producti/ely. They pointed out that, 
even though the information called for was available, the personnel and 
dollar costs associated with assembling it were substantial. They there- 
fore preferred to avoid assembling non-required irformation for which they 
could see no local utility. 

Specific Contments . 

• Table 1 (p. II-5) is confusing. 

• Probably teachers and certainly aides should be excluded from 
reviewing draft reports (last bullet. Table 6, p. III-7), It is 
difficult enough to get reports out in a timely manner without 
this type of input, 

© 'Parents and community members should certainly not be involved in 
conducting na.tive language testing. They are not trained, not 
unbiased, and often not literate (3rd bullet. Table 6, p. III-7). 

• School district --'cljninistrators should not review draft reports 
(last bullet. Table 6, p. III-8). 

• Emphasize developing an action plan for implementing recommended ' 
project improvements. Without an action plan, recommendations are 
likely to die (last bullet, p. III-IO). 

• Compiling all of the data called for in Table 7 (p. IV-3) is 
excessive. •'Tell us what we have to collect." Note also that 
some of the items can be derived from others and need not be sepa- 
rately compiled. 
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Text (from bottom of IV-2 to top of p* IV-8) is very difficult 
to follow because of tiie insertion of three lengthy tables* Note 
too that the only reference to Table 9 occurs in the middle of 
Table 8* 

Table 8 should include parent/community involvement activities and 
curriculum development activities • Note last sentence under Pro- 
ject Characteristics on p* IV- 2* 

The information called for in Table 8 (?• IV-6) that derives from 
requirements for grant applications is excessive • It would have 
to be collected each year as it becomes obsolete rapidly* 

Reconmendations to maintain close contact with project staff (last 
sentence of paragraph in middle of p* IV- 8) should be strongly 
emphasized* 

Figure 2 on p* IV-10 (really a Table) is useless* 

Regarding the first Self -Report example in Table 10 (p* IV-12) 
-•this is a cumbersome task -- should get this information from an 
interview* 

Table 11 (p* JVrl3) refers to "specified behaviors*" Clarify 
where these behaviors come from* 

Should note that behavioral checklist deals with individual behav- 
ior elements while coded behavior record deals with sequences of 
behaviors * 

Regarding recommendation that teachers not be informed as to when 
classroom observations will occur union agreement in Florida 
requires substantial advance notice. 



Pages V-1 and V-2 talk about language proficiency without defining 
it* Are we talking about oral language comprehension, oral lan- 
guage production, reading, writing, grammar, spelling, or what? 
Please provide some guidelines! 

Throughout Chapter VI, you should refer to means and medians, not 
just means. You told us earlier that medians would be preferable 
under certain conditions* 

Define scale score* 

Explain how to plot gap-reduction data a la Figure 3 (p, VI«3)* 

Florida adjtists for the regression-effect bias by multiplying the 
gain by the test-retest reliability of the pretest* Will this 
work in the gap-reduction design? 

The first sentence of the middle paragraph on p. VI-10 is confus- 
ing. 

First bullet under Planning the Testing Schedule -« what kind of 
exceptions are possible? Suppose a district wants to implement a 
spring-to-spring testing schedule. Vfhen should they pretest 
entering kindergartners? Is fall pretesting OK? Or does fall 
pretesting force the district into a fall-to-fall schedule from 
all grade levels? Can the kindergarten pretesting be skipped 
altogether? This is a problem with the regulations, -not the 
Users ^ Guide, but please provide some guidance. 

On p. VIII-3 you talk as if each school will have its ovm computer 
and do its own analyses. Do you not mean something else? 
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9 On page VIII-6, first paragraph under Other Summative Outcome Ana- 
lyses perhaps dropouts need to be .sorted by type, such as moved 
to another school in the district^ migrated seasonally and are 
expected to return, etc* for elementary grades there are very few 
dropouts in the sense that we tisually think of dropouts ♦ 

© The first sentence in the next paragraph needs to be explained* 

• On page VIII-7, second paragraph from the bottom, you are asking 
for too much especially since you acknowledge that none of this 
information is required by the regulations ♦ 

• Page VIII-8, exposure-to-treatment scores needs to be clarified* 

• The stratified analyses on p* VIII-IO are not required by the 
regs* You should acknowledge that fact* (The top of p* VIII-11 
presents that acknowledgement, but it should precede, not follow, 
the descriptions of the analyses)* 

Q Include teacher certification under implementation data (second 
bullet on p* IX-1). Are "waivered" teachers working toward certi- 
fication, etc.? Inclusion of a cultural component may also be 
important* 

There were no comments on Chapter X, and the site visitor discoturaged 
extensive comments on Chapter IX becatise a major restructuring of tTiat 
chapter was already underway* 

The Hay VjLsjt* The May visit was substantially different from that 
conducted in March* The major foctis was on technical issues associated 
with implementing the gap-reduction design* Considerable time was spent 
reviewing the conditions under which it is necessary to adjust for the 
regression-effect bias and the statistical procedures involved in making 
that adjustment* The Polk County evaltiation staff is also responsible for 
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Chapter 1 evaluation atid Is familiar with the regression-effect adjustment 
currently required by the Florida SEA. Unfortunately, the Florida proce- 
dure Is different from both the TIERS-recommended procedure and that 
reconmiended for the BEES. It Is also basically flawed In that It applies 
a multiplication adjustment to the gain rather than an additive adjustment 
to the selection/pretest mean score.* 

We reviewed the reason that the mean selection/pretest score Is nega- 
tively biased, the conditions tinder which an adjustment Is necessary, the 
factors that affect the magnitude of the bias, and the procedures that are 
recommended for removing it. We also discussed ways in which the gap- 
reduction model could be applied to non-test data (e.55. attendance rate). 
Because of the technical sophistication of the Polk Cotttity evaluator, some 
of our discussions were more in-depth than would have been appropriate 
elsewhere. 

Tvler. Texas 

Tyler was visited twice -- the first time on March 10 and 11 and the 
second on May 14 and 15, 1987. Tyler is a unique site in that it was just 
beginning the first year of its grant at the time the field test began. 
For that reason, it attempted to Implement nearly all of our recommend- 
ations as an integral part of its start-up activities. By the time of our 
first visit, the site had actually tried out most of the process-related 
ideas in the Users* - Guide > 



*The Impact of using the same set of scores for both selection and pretest 
purposes is that the pretest mean is spuriously low and the selected stu- 
dents will score higher on the next testing, even if there is no inter- 
vention. The problem with attempting to correct for this bias by apply- 
ing a multiplicative adjustment can be easily recognized if one considers 
the situation in which there is no pre-adjustment gain... With zero gain 
it follows that the adjustment must also be zero. On the other hand, the 
pretest score will be too low and should be adjusted upwards (thus creat- 
ing a negative gain) . 
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March Visit, Diiring the first visit we reviewed the Users* Guide 
page-by-page • Site personnel 'found that most of our recommendations could 
be implemented and said that they had found them helpful ♦ Some of the 
recommended practices they had planned to implement before they became 
involved in the field test, but there were others they could not have 
thougjit of on their own but did find helpful, 

A few of our recommendations conflicted with Texas law (e.g. modify- 
ing and/or translating tests and testing out of level) . We talked exten- 
sively during both visits (as well as at the Exit Conference) about this 
conflict without finding a satisfactory resolution for it. 

In-level testing is only a problem when students encounter the test's 
floor at pretest time. When that happens, however, "real" student gains 
will be \mderestimated by the gap-reduction design (and other evaluation 
designs as well) with the final result that the project will be made to 
appear less effective than it really is. 

The most tinfortxmate consequence of the requirement to test in level 
is that the greatest impact will be on projects serving the neediest stu- 
dents. The achievement growth of students participating in such projects 
will be the most seriously xmderestimated, and truly effective projects 
may even appear to have negative impacts. 

Specific comments follow. 

Specific Comments 

• 2nd paragraph on page 1-3 talks about what the gap-reduction 
design does not yield. Should say what it does yield. 

a 3rd paragraph on page II -1 -- Tyler has also involved parents in 
these preparatory activities. 
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« Table 1 on page II -5 is confusing and the examples are not help- 
ful* 

0 Table 2 (p, lll-2) contains too many activities in a difficult- to - 
follow sequence. Some grouping arrangement would be helpful* 

• There is no need to describe the project director as possibly 
"xinwilling or unable to direct the evaluation in detail." The 
early formation of a strong evaluation team is equally important 
even if the director is both willing and able* 

• The recommendations to involve aides in evaluation activities is a 
good one especially if the aides are teacher trainees* But 
neither they, nor parents and community members, nor district 
administrators should be involved in reviewing the draft evalua- 
tion report (Table 6, pp. III-7 and III-8). 

© The early pages of Chapter IV 'are poorly and inconsistently for- 
matted, titled, and punctuated. 

• The recommendation to keep student data bv student is good. 
Perhaps it should be emphasized more strongly. 

• The bottom paragraph of p. IV-2 should be moved to the top of p. 
IV-8. 

• Too much documentation of pedagogical methods, materials, and 
techniques are called for. Could this be limited to critical fea- 
tures? 

B Educational and professional qualifications should be doctimented 
for all teachers and aides not just sturanarized across the project* 
This is very important! (Table 8, p. IV-6). 
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The information called for on parent advisory coimsels and staff 
training activities is appropriate • We collect it and find it 
useful. (Table 9, p. IV-6) 

Table 9 (p. IV-7) is not referenced in the text. 

Figure 2 on p. IV-10 is not a data extraction form, it is a sample 
of extracted (and useless) data. 

Tyler U3es all three types of self-report measures and finds them 
useful (p. IV- 10). 

The recommendations on classroom observations (p. IV-11) are too 
non- directive. Tyler wants more specific guidance on what they 
should (or are required to) do. 

With regard to delayed report observations, how long a delay is 
recommended, permissible? (p. IV-11) 

Table 11 (p. IV-13) should specify behaviors - who "specifies" 
them and how? etc. 

Step 4 (P. IV-15) is helpful specific guidance for classroom 
observation, but information should be expanded and presented ear- 
lier (see 2 bullets above) . 

Page V-7. Tyler cannot make any modification to standardized 
achievement tests (although they can modify instructions). 
Extending time limits, simplifying language, and translating tests 
are prohibited by Texas regulations. 

Tyler collected all of the student characteristics data called for 
in Table 13 because of our recommendations. Several suggestions 
were made regarding how redundancy could be reduced, (page VII-7) 




« Use of raw scores is discouraged (Top paragraph on p* VIII-2). 
The reasons are not made clear. What is gained by making "diffi- 
cult" score conversions? 

• Tyler needs help with data base management* Users ^ Guide recom- 
mendations are inadeqtxate. Could we recommend specific software 
'or source of technical assistance? (p. VIII-2, bottom two para- 
graphs) . 

0 Page VIII-3 implies that analyses are done at the school level. 
All of Tyler's analyses are done in the district office by dis- 
trict staff. Is this contrary to the Users' Guide recomment^ 
ations? 

9 The I?s?y?/,. G^t<?e says that "other sunmiative analyses" need bt done 
only for project participants (p. VIII-o, near top of page). The 
recommendation is unclear. What about former participants? What 
about comparison groups? 

o What do you mean by a "trend analyses?" (last line on p. VIII-6) . 
Please clarify* 

• Please emphasize more strongly that transitional bilingual pro- 
grams cannot be adequately evaluated on a one-year basis. A lon- 
gitudinal approach is needed (p. VIII- 13, third paragraph). 

• The next paragraph (p. VIII- 14, fourth paragraph) should include 
an example. 

Tyler had no comments on Chapter X. Comments on Chapter IX were dis- 
couraged since a major revision of that chapter is currently tinderway. 

» 

MajLVisiJE. At the time of the May visit, Tyler was beginning to 
think about the outcome component of their evaluation and requested tech- 
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nical assistance on various aspects of the gap -reduction design. Of pri- 
mary interest vas the fact that the design could be applied to non-test 
data* We worked through examples tising classroom grades (tiseful particu- 
larly for assessing the performance of reclassified LEPS) and attendance 
data* 

Site personnel were shown that a simpler form of the design (one that 
does not require calculating RGI's or even standardizing growth estimates) 
can be useful for local-level project improvement. Since these topics are 
covered only superficially in the Users' Guide it was agreed that the3' 
should be explained in greater detail when that document is next revised* 

We also discussed the origin and cause of the regression-effect bias 
and how to correct for it. Persons with evaluation responsibilities for 
programs other than Title VII attended parts of these technical assistance 
sessions « 

The topics of functional- level testing and modifying tests to 
"enhance their psychometric integrity" were also discussed, but little 
progress was made toward resolving the conflict between Users' Guide 
recommendations and Texas regulations. 

Valley View. TX. The Valley View school district in Pharr, Texas, 
was visited on April 7 and 8, and again on June 5th* Unfortunately, 
between the initial meeting of the field-test sites in San Francisco dur- 
ing the first week of October, 1986, and the first site visit, major 
administrative changes had occurred within the district. Both the Dis- 
trict Superintendent and the Coordinator of Federal Programs were new. 
The external evaluator, with whom we had expected to work closely, had 
also been replaced* Not surprisingly, the turmoil caiised by these major 
changes resulted in field- test activities being given a low priority. 
Still, the site was anxious to collaborate and to do what it could to 
assist us in meeting our objectives. 
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Valley View was unique among our sites for other reasons as well. 
Some 95% of its students are Hispanic LEPs and approximately the same per- 
centage are poor (participating in the free-lunch program) . i^ll students 
in the district receive bilingual education services except for about 5% 

those students whose parents want them to receive no instruction in 
their native language. There is no mainstream in the sense that that term 
usually connotes* 

In addition to those demographic characteristics already mentioned, 
Valley View is a very poor school district ^the local tax base is minimal) 
that serves a rural and highly mobile population. Most of its students do 
not pursue their education beyond the ninth grade, and there is no high 
school in the district. 

The district does not use a test-scoring senrice -- teachers score 
the tests. It has a non- functional microcomputer. All records are kept in 
paper files stored in boxes, and all "number crunching" is done by the 
external evaluator using his own hardware and software. 

Th^-ApyU v^sj.t. The Federal Programs Coordinator was the only 
district person who participated in our site visit. The external evalu- 
ator was present only briefly the first day. A breakfast meeting was held 
with him the second day -- largely on his own (unpaid) time. 

It was clear that evalxiaticn resources in the district were extremely 
limited.- The external evalxiatov's $2,500 contract required him to do all 
of the statistical data analysis plus six days worth of observJ.ng class- 
rooms and providing feedback to project teachers. The district did not 
wish to alter this plan to allow tiii-e for reviewing the Users ^ Guide and 
participating in our meetings. 

The Federal Programs Coordinator had read the Users ^ Guide prior to 
the March meeting. He claimed to agree with all of its recommendations 
although there were many he could not implement because of limited 
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resources and others that conflicted with Texas laws/regulations. He 
offered no suggestions for improving the guide. 

He was particularly interested in the Users' Guide reconanendations 
that curricular validity be a major factor in selecting tests to measure 
achievement growth* He claimed that the instruments xised in Valley View 
were chosen because they were used in neighboring districts even though 
there were major differences between the districts' curricula • In think- 
ing about the currently used tests, he felt that they might have low cur- 
ricular validity. He indicated he would give high priority to reviewing 
alternative instrjunents for possible use next year. 

We also discussed the problems associated with functional -level test- 
ing in Texas* Given the in- level testing requirement, it was mentioned 
that looking for tests with low floors as well as a good curriculum match 
might be a useful strategy. 

No other significant issues were discussed during the March meeting. 

The, June Meeting. The June meeting was conducted after school Lid 
already closed* Only the Director of Federal Programs was available for 
discussion. Again, he had no specific comments on the Users' Guide . 

We discussed the fact that Valley View's external evaluator would be 
implementing a TIEftS Model A evaluation rather than a gap- reduction 
design. The RMC site visitor then explained that the two designs were 
closely related and that the former could be converted to the later with 
minimal computations on a hand calculator. The procedures were worked out 
and a step-by-step computational routine was developed. RMC also offered 
to provide technical assistance in making the conversion. 

The discussion then turned to the "fairness" of tising national norms 
as a comparison group for Valley View students. Pointing out that the 
comparison was neither fair nor unfair as long as no assumption was made 
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as to how well the project students would do without the project did 
little to convince the Federal Project Coordinator. 

He would only feel comfortable using a local, mainstream comparison 
group and there is no such group in Valley View. 

We discussed other possible comparison groups, but without reaching a 
satisfactory resolution. 

Additional time was spent going over the curricular validity issue. 
This discussion appeared to be useful for reinforcing the conclusions that 
had been tentatively drawn during the previous visit, but no new ideas 
were introduced. 

At the end of the first day, there appeared ho nothing left to 
discuss. The site visit was thus concluded one day ahead of schedule. 

Valley Ce nter. Calif omi» 



During the first visit to Valley Center, activities focused on ana- 
lyzing the project and evaluation designs. The analysts revealed that 
project objectives needed clarification, and two concerns about the objec- 
tives were identified. 

First, student outcome objectives vised grade -equivalent scores as 
growth measures. Use of these scores, while appealing to practitioners 
and laymen, is psychometrically unsound. Project staff asked whether 
expected gains expressed in grade -equivalent scores can be converted to 
another metric such as NCEs. The Users' Guide should address this issue, 
as well as providing guidance on how to write student outcome objectives 
that reflect the gap-reduction design. 

Second, curriculum- specific objectives are difficult to write. They 
tend to be confused with specific expectations, benchmarks, or skills, 
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such as "find the sum of two numbers A curriculum-specific outcome 
objective should include a reference to certain instructional activities 
and/or materials which are (xisually) designed to teach several skills. In 
addition, such an objective should specify the method of assessment (such 
as a imlt test or checklist of skills) and the criterion for success • The 
users ^ Guide needs to make this clear. 



In analyzing the project design, it became apparent that different 
types of students having different degrees of English proficiency receive 
different types of instruction (see Appendix H) . After an initial meeting 
with the project director, a meeting was held in which all project staff 
could discuss instructional design, what teachers should be doing in their 
classrooms, and other critical features of the project. While some ini- 
tial thoughts were formulated, some confxision about tlie design of the pro- 
ject remained. 



During his second visit to this site. Lam visited project classrooms 
and met again with project staff. - Son after this meeting, the site devel- 
oped a list of global and specific critical features. This list, included 
in Appendix G, closely follows Users ^ Guide recommendations. In generat- 
ing the list, project staff expressed confusion about whether critical 
features were only instructional features or whether activities such as 
district support, parent involvement, or project management should be 
included. The Users ^ Guide should explain -critical features more fully. 

Based on the list of critical features, project staff have begwi to 
develop a classroom observation instrument. This instrument, included in 
Appendix G, is not yet complete. As recommended by the BEES, project 
staff also generated evaluation questions and identified problems in pro- 
gram implementation. 
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In reviewing tests to determine their curricular validity, the pro- 
ject staff decided to review test objectives rather than individual items 
as recommended by the BEES. Project staff feel reviewing test objectives 



34 



I 



is Jiore efficient • Their experience suggests that the BEES might include 
a section on matching test and curricular objectives. 

Project teachers reviewed the CTBS, identifying for each test objec- 
tive whether or not they teach that objective at their grades. They also 
rate each objective in terms of the importance of developing the skills, 
the number of skills covered, and the amount of time needed to teach the 
skills (see Appendix H). We will examine these data to compare the ade- 
quacy of this objectives approach to the currently recommended item-by- 
item approach* 

Data management is also a probJ^m for Valley Center. Most data are 
maintained in separate paper files and have not been combined onto a cen- 
tral student form. Future efforts will be needed to create a computerized 
data management system. The U sers^ Guide should provide more g^iidance on 
computer software for data managec^ent. 

San Jose Unified S chool District 

Because of the proximity of the district to the RMC office, four, 
instead of two site visits were made to San Jose. 

• 

While there are two Title VII projects in the San Jose School Dis- 
trict, only one of them, the Vietnamese and Other Indochinese Curricultun 
and Educational Services (VOICES), participated fully in the pilot test- 
ing* 

During the first two visits to the district, the district evaluation 
and testing coordinator was prdsent at the group meetings. The first con- 
cern the j oject staff had was the kind of data the project should report 
for evaliiation (The Gn^ie only describes what data to collect) . 

A liat of data to report was subsequently generated and included in 
the later version of The Guide . 
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In the first two meetings, the project and evaluation designs were 
discixssed* Dxiring that time, the project's needs assessment data, objec- 
tives, and instructional design were examined, clarified, and elaborated 
where necessary. Since the project was in its third year, there was some 
reluctance to make too many changes to the original designs • However, it 
was agreed that the project's actual instructional features should be 
identified and classroom observations should be conducted to monitor and 
determine the extent to which these desired Instructional feattxres are 
actually being implemented in the classrooms « 

Through brainstorming among Lam and the project staff, some examples 
of specific critical instructional features were identified. Two of these 
features are: (a) aide summarizes, using U, the content taught by teach- 
ers, and (b) students are grouped by reading ability* Using all of the 
examples, project staff were supposed to develop a checklist of specific 
critical instructional features for classroom observations. However, due 
to time and resource constraints, thus far the project has developed only 
a general observation form (see Appendix H)* If the project receives an 
extension for two more years, a more specific observation form tailored 
for the project, which the project staff feel is important to have, will 
be needed to document program implementation. 

A major component of the VOICES project is materials development. 
The project has been developing Vietnamese language arts and reading mate- 
rials (programs) and social studies materials in both English and Vietna- 
mese for kindergarten through fifth grade. ' For the VOICES project, the 
adequacy of the materials developed by the project is determined by 
examining student performance on the CTBS. Whatsis missing is the 
intermediate step of reviewing, the material for quality of construction 
and appropriateness. What the project staff found lacking in the Guide 
are procedures for evaluating materials development efforts. There is no 
guide on how to evaluate the quality of instructional materials either 
commercially or locally developed. 
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The VOICES project staff find curriculum-specific objectives easier 
to develop and more appropriate than do staffs of other projects since all 
VOICES teachers use the same curriculum, and there are criterion- 
referenced tests developed to accompany the curriculum. However, it is 
still not clear whether or not curriculum- specific objectives are needed 
if student skill areas and cxurricula are specified in advance and verified 
via classroom observation, and data on studexit classroom performance are 
collected. Writing curriculum-specific objectives may be an extra burden 
that can be avoided in this project. 

Since the project receives adequate assistance from the district 
evaluation and testing office in data collection and management, not much 
technical assistance was given in this area. 

During the last site visit. Lam visited a kindergarten class to 
gather first-hand information regarding classroom processes. Based on 
this observation, he was able to provide suggestions for improving the 
draff classroom observation form. 

A problem encountered by the VOICES project that has not been dealt 
with in the Users ^ Guide is a concern about estimating student growth 
across time when the test is changed. For the VOICES project, there will 
be a change from a commercial test to a test tailor-made for the San Jose 
School District. How one should compute RGI's with different tests given 
at different points in time should be discussed in the Guide . 

New York Community School District ^5 

There are two Title VII projects in New York Community School Dis- 
trict ^5. However, most of the pilot testing effort has been concentrated 
on the Critical Thinking Skills (CTS) project, which began in October of 
1986. The CTS project is designed to enhance critical thinking skills of 
LEP students in grades K through 3 by instilling critical thirking teach- 
ing strategies into the bilingual classrooms. Teacher training and the 
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provision of teaching aides are the activities tised to produce these 
results* 

Lam worked with the project director, resource teacher, bilingual 
program coordinator, and external evaliiator on ensiaring the project's 
evaluability. The project objectives were completely revised in accor- 
dance with Users' Guide recommendations* The philosophy and theoretical 
framework underlying tlie teaching of critical thinking skills is very well 
defined. The challenge the project faces is to translate' the theory into 
classroom practices, implement these practices, and then measure student 
growth in critical thinking skills. 

Through a great deal of discrission, the project developed a classroom 
observation checklist that covers thinking skills teaching strategies and 
methodology for teachers and aides, and classroom management, environment, 
and interaction (see Appendix J). In the coming year, the checklist will 
be pilot, tested and then used to monitor and document the level of program' 
instructional service delivery. The initial data collected can. also be 
tised to determine areas in which teachers or aides should receive train- 
ing. This point is not mentioned in the current Users' Guide . As do 
instruments developed in other pilot sites, the observation instrument • 
requires trained observers who are familiar with bilingual and ESL metho- 
dologies. 

Another problem in teaching critical thinking skills is to assess 
student growth in these skills. The project staff has been searching for 
a satisfactory test, but has not found one. Consequently, the project is 
now developing its own test, which is against the reconmendation of the 
BEES. Even if the quality of the test is acceptable, the project will 
have to administer the test to a comparison group in order to implement 
the gap-reduction analysis. Administering any test to any students ir. New 
York City for the sake of obtaining comparison data' for a Title VII pro- 
ject may present some legal problems. This point should be acknowledged 
in tile Users' Guide . 
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The project has been diligently collecting student data as recom- 
mended by the BEES, xising the sample forms in the Volume Two of the gsers* 
Gui^^ Currently, data have been collected from all Title VII a^.tudents* 
The next step is to entar these data into the computer • Creating a com- 
puterized data base is an area in which the project needs technical assis- 
tance ♦ 

As in the other New York site, student mobility is a major problem 
for obtaining a representative sample of data at this site^ 

New York Community School District 410 ' 

Project CARE (Career Awareness Resource In Education) is a project in 
its first year. It is designed to infuse ESL methodology and career edu- 
cation into regular bilingual classes through teacher training and parent 
involvement and education* In an initial meeting with the project direc- 
tor and her supervisor, a great deal of time was sp^nt discvissing what the 
program consists of and how best to document its implementation. 

The program. focuses on career education for students ard on the 
enhancement of teachers' ESL sW.lls. Therefore, the evaluation staff 
needed to develop a checklist of critical featmres for these two areas, 
/tn ESL checklist was adapted from a district checklist, but it needs revi- 
sion»to make it easier to implem;.nt (see Appendix I). A checklist for ' 
career edur .::ion has not yet been developed. 

• • 

During the two visits to this project, project objectives were clari- 
fied and revised. There was also a discussion of how the project should 
measure changes iit students' self concepts. Project staff were not happy 
with available paper and pencil measures. Lam suggssted that project 
staff follow Users' Guide recommendations to use indirect indicators such 
as absenteeism or participation in extra-curricular activities. There is 
some question about whether one should expect the activities of this pro- 
ject to produce improved self concepts. The Users' Guide should have a 



ERIC 



39 

44 



clearer disctission of the need for a logical linkage between treatment and 
anticipated outcomes* 

The project also has a problem with high attrition • The posttest 
might include a completely different group of students from tOie pretest. 
The Useyc^ Guide should address analysis issues related to high student 
mobility. The suggestion has been made that attrition at lower grade lev- 
els is more random than in the ttpper grades, so that systematic bias in 
analyses should be less at those grades. 

New York state prohibits parents from observing classrooms, so the 
Users ^ guid<5 recommendation is inappropriate for this project. It is also 
not possible in New York for teachers to review tests item by item, since 
they are prohibited from seeing the tests except on the dates they are 
being administered. 
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Ill- SUMMARY OF CHAPTER-BY-CHAPTER REVIEW OF USERS ^ GUIDE 
Comments from fll Particlpair^ts 



Chapter I. TntroductiPn 

• The chapter should include a better description of the BEES, including 
its limitations* 

e The chapter should make clear who the BEES is designed for and who is 
the audience for the Users ^ Guide . 

• The chapter should include some discussion of previous research. 
Chapter II, Ensuring the Project* s Evaluabi 7 j tn^ 

© Weeds assessment. The discussion is too technical. Give an example of 
a good needs assessment. Don't use a question format. If- mobility is 
high, these data may not be appropriate baseline data* Need more dis- 
cussion of baseline data. 

• Qbycttves. Drop cxirriculum specific objectives. Define what objec- 
tives are needed to conform to what regulations. Realistic criteria 
are very difficult to determine. 

9 General . The chapter"needs to be more definite. The format is too 
busy. The chapter needs a glossary, or definitions in the text. More 
examples and a clearer format are needed. 
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Chapter III, Plan ning the Evaluation 

• It is difficult to form an evaluation team it is not always possible 
to involve other staff* 

• It may not be possible to involve parents in an evaltiation* 

• This chapter should tell what implementation difficulties really are 
--use realistic examples. 

o There are too many tables in this chapter they break up the text. 
Chapte r IV, Documentinp Program Processes 

• Tell exactly what is required by the regulations and what is an inter- 
pretation or recommendation by the authors of the Users' Guide. 

• The Users' Quide should make clear that self reports should not be used 
without corroborating information. 

• The examples in this chapter are poor. Use positive, not negative 
examples. 

• Teachers' language fluency is very hard to assess. 

• Other information would be more useful to collect than the information 
listed. 

e Discuss who should conduct classroom observations. 

• Many teachers will not agree to classroom observations. 
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• Differentiate betweeri conducting classroom observations for teacher 
evaluation and for program evaluation* 

• The Guide should make clear that a "delayed" report observation is only 
delayed until the observer has left the classroom* 

• The data recoaanended for collection on students, instructional services 
and teachers should be prioritized. 

As discussed previously, two questionnaires were administered, one 
prior to the June workshop, which addressed the availabUity and accessi- 
bility of these data. The second questionnaire was administered during 
the June workshop and it addressed the usefulness of these data for esti- 
mating program effectiveness^. Table 4 and Table 5 suamarize results from 
these two questionnaires. As can be seen in Table 4, the information that 
has more "not available" responses than "available" responses include 
years of education missed, dates entered and exited other settings, 
enrollment in postsecondary education institution, and minutes per week 
devoted to specific learning activities. 

In Table 5, it can be seen that information regarding whether or not 
the student is enrolled in public or private school and sex appear to have 
little importance for estimating program effectiveness. Other information 
that has questionable importance include date began U.S. residence and 
date of birth. All other data listed in the survey are perceived to be 
important or very important for estimating program effectiveness. 



Due to the similarity in response to the question regarding the use- 
fulness of the data for estimating program effectiveness and the question 
regarding the usefulness of the data for improving program performance, 
only the responses to the first question were summarized 
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TABLE 4 

BTaluatloo Aa stgtaneg Center-Weat 
Data CoUee tlon and Bftpprtlni^ Survey 



I« SniDKWT IMPOMATTn w 

A. Background T nfonMtlon 



Infocaatlon 
Available? 

-L JL ^ OTHER 
(Z) 



If Kot Available, 
Uow Accessible? 



uace 01 birth 


89 


0 


11 


0 


0 


rxace or birth 


100 


0 


0 


0 


0 


bctinicity/language group 


100 


0 


0 


0 


0 


Date began U.S. residence (date) 


44 


22 


22 


11 


2.5 


Language used in .home 


100 


0 


0 


0 


0 


Fluency in English as of (date) 


100 


0 


0 


0 


0 


Fluency in native language (LI) 
if other than English as of (date) 


89 


0 


0 


11 


0 


Years of education conpleted as of (date) 


67 


33 


0 


0 


1.7 


Years of education oissed as of (date) 


33 


44 


22 


0 


1.7 


Family income status (e.g., AFDC, 
participation in National School 
Lunch Program) 


67 


33 


0 


0 


3 


Public or private school 


78. 


11 


0. 


0 


4 


School grade level 


100 


0 


0 


0 


0 


B. Service Data 












Dates entered and exited project 


89 


11 


0 


0 


5 


Other instructional settings (e.g., 
oainstream class. Chapter 1 program, 
special education, migrant progran, 
ESL program) 


89 . 


11 


0 


0 


3 


Dates entered and exited other settings 


22 


67 


11 


0 


2.5 



0 
0 
0 
2 
0 
0 

0 
1 
1 

1 
4 
0 



0 
0 
0 
3 
0 
0 

0 
2 
2 

5 
4 
0 



0 
0 
0 
2 
0 
0 

0 
3 
3 

3 
1 

0 
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3 
1 



3 
4 



1 
6 



C» Won-Te«t Data 

Years retained In grade as of (date) 
Whether or not student dropped out 
Days absent froa project dassrooa as 
t ot'l^tTatT'"^ -chool days possible 

Date of special education referral 
or placeaent 

Date of placeaent In gifted and 
talented prograa 

Enrollment In postsecondary 
education Institution 

Setting froa which entered project 

Grade point average prior to entry 
in project 

Setting to which exited froa project ' 
Grade point average after exit 
D« Foraer Project SttiH«.n»« j 
Standardised achieveaent test data 
Classrooa performances (e.g., grade) 

II. INSTRUCTION 

Acadeaic and non-academic subjects 
taught in the project 

Method of service pfovision (e.g., 
tutoring, dassrooa instruction) 

The following variables apply to each class or 

English and LI materials and how thsv 
are used ^ 

Percenta of instruction conducted in LI 
and in English ' 



Inforaation 
Available? 

JL -M, M. OTHER 
(%) 



89 11 

67 33 

100 0 

89 11 

67 " 22 



78 22 
89 11 



100 0 

89 0 

78 22 
67 33 



ERIC 



45 



0 
0 



0 
0 



11 
0 

0 
0 
11 



0 
0 



0 0 

0 11 

0 0 

0 0 



If Not Available, 
Uow Aecessibl:^? 



1 

2.3 



11 


56 


11 


22 


1.6 


1 


3 


5 


67 


22 


11 


0 


1 


1 


1 


2 


33 


33 


11 


22 


3.7 


2 


5 


3 


44 


44 


11 


0 


2 


1 


4 


4 


22 


22 


0 


11 


3.5 


2 


5 


4 
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1 

2 

0 
2 
2 



3 
5 



3 
5 



1 1 
3 3 

0 0 

2 1 
2 1 



3 2 
5 1 



0 0 0 0 

0 0 0 0 

3 3 3 2 

2.3 1 3 3 



Pattern of dassrooa language uae (e.g. . 
concurrent, preview/review, or eltemWng: 

•treteglea Including 
inatruction and Mcageiient procedureiT 

••S^^f Hr.'^.*."''* teaching behavior., 
of content fro. atudenta' hoae cultirej 

Locations of inatriiction (e.g., malnatreaa 
daaaroom, apadal biUngual claaaroo.! 
newcomer center, reaource rooa) 

Student groupings (e.g., LEP students are 
grouped by EngUsh proficiency, all 2e? 
students are together all day) 

Student/teacher ratio, studens/alde 
ratio per class 

Variability of students within a 
^uS^^Jt^';^" educational 
ibluS^n* ^"S'^'Se;* language 

Function of aides in dassrooa 
C* instruction, X Banageaent) 

Form ox instructional group (e.g.. 
whole class, tutorial) 

each aub.lect area taught, ainutea per 
week devoted" to apecTfic sctivitleneJ^. , 
jath workbooks, English spelUng, laigufgi 



Inforaation 
Mailable? 



67 33 0 



56 44 0 

^00 0 0 

78 22 0 

100 0 0 

89 11 0 

57 33 0 

89 11 0 

44 56 0 



III. STAFF 



Academic preparation of each staff 
tnomber, including: 

• level of education, 

• academic preparation In field of 
education, 

• credentials and certiflcatea, and 

• academic r reparation in field of 
bilingual/ blcultural education 



78 22 0 

67 22 11 

89 11 0 

67 22 11 



If Not AvaUab]«, 
How Accessible? 



O.TPR 



0 
0 
0 



mm MBLijifli ^ 

„ 3 2 A 3 



3 2 4 4 

0 0 0 0 

2.5 2 3 2 

0 0 0 0 

3 3 3 1 
3.7 2 5 3 

4 4 4 1 
1.7 1 3 4 



3.5 2 5 2 

3.5 2 5 2 

2 2 2 1 

3.5 2 5 2 
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• In UUngual cUgatooaa 



it!^''T of each ctaff aeaber 

8SoSJ*"«.^°««*8es understood and ' 

• LI 



Xnfonuitlon 
Availablet 



-2 JL 2K OTHER 



67 33 0 0 
67 33 0 0 



55 44 0 0 

56 44 0 0 



If Not Available » 
How Aeeasaible? 



3.3 2 
3.3 2 



2.7 2 

2 4 



5 3 
5 3 



3 4 
3 4 



ERIC 



52 



TABLE 5 



STUDENT, INSTRUCTION, AND STAFF INFORMATION 
(Response to question: "How important is information 
for estimating program effectiveness?") 



I. STUDENT INFORMATION 



A. Background Information 


Meanl SD^ 






Max 


1 


2 




L 


5 
















(Percent) 




Date of birth 


3.4 


1 1 


9 


2 


5 


22 


22 


22 


22 


22 


Place of birth 


3.6 


1.4 


9 


1 


5 


11 


11 


22 


22 


33 


Ethnicity/language group 


4.2 


1.1 


9 


2 


5 


0 


11 


11 


22 


56 


Date began U.S. residence 


3.1 


1.8 


9 


1 


5 


n 


22 


33 


11 


22 


Language used in home 


4.8 


0.7 


9 


3 


5 


0 


0 


11 


0 


89 


Fluency in English as of 


4.8 


0.7 


9 


3 


5 


0 


0 


11 


0 


89 


Fluency in native language (LI) 
if other than English as of 


4.8 


0.7 


9 


3 


5 


0 


0 


11 


0 


89 


Years of education completed 
as of 


4.4 


0.7 


9 


3 


5 


0 


0 


11 


33 




Years of education jwissed as of 


4.6 


0.7 


9 


3 


5 


0 


0 


11 


22 


67 


Family income status (e.g., 
AFDC, participation in National 
School Lunch Program) 


3.7 


0.9 


9 


3 


5 


0 


0 


i»5 


22 


22 


Public or private school 


2.6 


0.9 


8 


1 


4 


13 


25 


50 


13 


0 


School grade -^l 


4.4 


0f9 


9 


3 


5 


0 


0 


22 


11 


67 


Sex 


2.6 


1.4 


9 


1 


5 


33 


11 


33 


11 


11 


B. J>ervice B&ta 






















Dates entered and exited project 


4.9 


0.3 


9 


4 


5 


0 


0 


0 


11 


89 


Other instructional settings 
(e.g., mainstream class. Chapter 
1 program, special education, 
migrant program, ESL program) 


4.7 


0.7 


9 


3 


5 


0 


0 


11 


11 


78 


Dates entered and exited other 
settings 


4.1 


0.9 


9 


3 


5 


0 


0 


33 


22 


44 
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C. NoT^-Te^t p»ta Heani sD ^ N M^r, m»v i 2 ^.^ 

(Percent) 

Years retained in grades as of 4.3 .9 9 3 5 .0 0 22 22 56 

Whether or not dropped out 4.7 .7 9 3 5 0 0 11 11 78 

Days absent from project class- 4.4 .7 9 3 5 0 0 11 "i^ 
room as of and total school 
days possible as of that date 

Date of special education 4. 31. 09 2 5 0 11 0 33 Sfi 

referral or placement u ii 0 i3 56 

Date of placement in gifted 3.9 1.3 8 2 5 0 PS 0 " -^R ^« 

and talented program u u J8 38 

Enrollment ^in postsecondary 3.7 1.7 9 1 5 22 0 11 22 44 

education institution u ii 22 44 

Setting from which entered 4.0 1.5 9 1 5 11 11 0 90 

project " " 

'?rade point average prior to 3.6 1.2 9 1 5 11 0 33 33 22 

ntry in project "^"^ 

Setting to which exited from 3.8 1.5 9 1 5 11 11 11 72 

project "^"^ '-^ 

Grade point average after exit 4.2 .8 9 3 5 0 L 22 33 44 
D. Former Pro ject Studenftg 

Standardized achievement test 4.9 .39 4 5 0 0 0 1189 

^^^^^room performance (e.g., ' 4.7 .5 9 4 5 0 0 0 33 67 
II. INSTRUCTION 

Academic and non-academic 4.9 .39 4 5 0 0 On«Q 
subjects taught in the project 

Method of service provision 4.9 .3 9 4 5 0 0 0 11 89 

(e.g., tutoring, classroom 

instruction) 
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The following variables apply to each class 
or subject matter offered by project: 



MeanLsfi2. N___Min Max 1^345 

(Percent) 

English and LI materials and 4*71. 09 2 5 Oil 0 0 89 

how they are tised 

Percents of instruction 5.00 9 5 5 00001 00 

conducted in LI and in English 

Pattern of classroom language 4^6 .7 9 3 5 0 0 11 22 67 

use (e.g., concurrent, preview/ 
review, or alternating) 

Critical teaching strategies 4.8. 49 4 5 000 99 7R 

including instruction and manage- ^ /o 

ment procedures (e.g* , use of 

active teaching behaviors, use 

of content from students' home 

culture) 

Locations of instruction (e.g., 4^2 .9 9 2 5 0 11 0 44 44 

mainstream classroom, special 
bilingual classroom, newcomer * 
center, resource room) 

Student groupings (e.g., LEP 4. 7 . 59 4 5 000 33 67 

students are grouped by English 
proficiency, all LEP students 
are together all day) 

Student/teacher ratio, student/ 4.6 .7 9 3 5 0 0 11 99 /;? 

aide. ratio per class u ii b/ 

Variability of students within 4.7 .7 9 3 5 0 0 11 11 78 

a classroom (e.g., age, grade, 
educational attainment, home 
language, language abilities) 

Function of aides in classroom 4,7 .7 9 3 5 0 0 11 11 78 

(% instruction, % management) 

Form of instructional group 4.9 .3 9 4 5 0 0 0 11 89 

(e.g., whole class, tutorial) 

For each subject area taught, 3.6 1.2 8 1 5 22 11 11 0 44 

minutes per week devoted to 

specific activities (e.g., 

math workbooks, English spelling, 

language lab) 
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Ill- STAFF 



Academic preparation of each KeavX SD ^ N Min Max 1 3 4 5 

staff member, including: . . (Percent) 

• level of education, 4*4 .9 9 3 5 0 0 22 11 67 

a academic preparation in 4*8 .7 9 3 5 0 0 11 0 89 
field of education, 

• credentials and certificates, 4.7 ^7 9 3 5 0 0 11 11 78 
and 



0 100 



• academic preparation in field 5. 00 9 5 5 00 
of bilingual/bicultural 
education 

Teaching experience of each staff 
member, including: 

o years of experience in 4.3 1.0 9 3 5 0 0 33 0 67 

monolingual classrooms, and 

© in bilingual classrooms 4.8 ,49 4 5 0 0 0 22 78 

Language abilities of each staff 
member, including, languages under- 
stood and spoken and degree of 
fluency in: 

© English 5. 00 9 5 5 0000 100 

LI 5.0 0 9 5 5 0 0 0 

^ l-=least important; 5«most important 
^ SD-Standard deviation 



0 100 
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Table 6 sxumnarizes results from the questioimaire designed to obtain 
input from the field site. staff in terms of accessibility and usefulness 
of the project backgrotand information recommended by the BEES* As can be 
seen in the table, the only two community characteristics that are per- 
ceived as accessible and useful are percentage of residents below federal 
poverty level and ethnic makeup of coinmunity* All the data regarding dis- 
trict demographics and school characteristics are considered accessible 
and useful* 

Chapter V, — Selecting/Adapting /Developin^ Instruments for AssessiTig 
Suironative Outcome Ob-jectives 

o It is very difficult to find curriculum- relevant standardized tests* It 
is also hard to get a group to review tests item-by- item* 

• Why not write a test in LI or translate a test to LI? If there are good 
reasons, they should be spelled out. 

• Why use reliability as a test selection criterion when all standardized 
tests have siwilar reliability? 

© Why use cultural bias as a test selection criterion when it is not 
important for estimating growth? 

• Give examples of cultural bias* (An example was suggested*) 

• Telling new teachers to teach test-t king skills can catise problems* 

Chapter VI. Im plementing the Gap-Reduction Design 

e This chapter is too easy for evaluators and too hard for practitioners* 
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TABLE 6 



Project Background Information 

Accessibility 



Mean-*- SD 



N 



Hin Max 



Usefulness 
Mean2 SD N Hin 



Max 



A* COMMUNITY CHARACTERISTICS 

Percentage of residents below federal poverty level 

Unemployment rate 

Workforce composition 

Ethnic makeup of community 

Average length of residence in community 

Percentage of single-parent families 

Educational level of adult community residents 

Community attitude toward bilingual education 

B- DISTRICT DEMOGRAPHICS 

Approximate peak student enrollment 
Number of schools in district 
Characterization of school district 

(rural, inner city) 
Number of classrooms in bilingual program 
Percentage of students eligible, for free lunch 
Ethnic makeup of student body 

C* SCHOOL CHARACTERISTICS 
Student enrollment 

Percentage of students eligible for free Ixinch 
Ethnic makeup of student body 
Number of bilingual teachers 
Number of bilingual aides 
Number of bilingual classes 



• 1 — not accessible; 5 - very accessible 
^ - 2 - not useful; 5 « very useful 



3.4 


1.6 


9 


1 


5 


4.2 


1.3 


9 




5 


2.7 


1.6 


9 


1 


5 


2.8 


1.6 


9 




5 


2.2 


1.0 


9 


1 


3 


2.9 


1.4 


9 




5 


4.1 


1.2 


9 


2 


5 


4.0 


1.3 


9 




5 


2.0 


1.1 


9 


1 


4 


?.l 


1.5 


9 




5 


1.9 


.9 


9 


1 


3 


3.1 


1.3 


8 




5 


1.3 


.7 


9 


1 


3 


2.8 


1.7 


9 




5 


2.1 


1.1 


9 


1 


4 


3.9 


.8 


9 


3 


5 



4.8 
4.9 
4.9 

4.9 
4.7 
4.6 



4.9 
4.8 
4.6 
4.9 
4.8 
4.9 



.4 
.3 
.3 

.3 
.5 
.7 



.3 
.4 
.7 
.3 
.7 
.3 



9 
9 
9 

9 
9 
9 



9 
9 
9 
9 
9 
9 



4 
4 
4 

4 
4 
3 



4 
4 
3 
4 
3 
4 



5 
5 
5 

5 
5 
5 



5 
5 
5 
5 
5 
5 



4.2 


1.0 


9 


2 


5 


4.2 


1.0 


9 


3 


5 


4.6 


.7 


9 


3 


5 


4.9 


.3 


9 


4 


5 


4.4 


1.0 


9 


2 


5 


4.7 


.7 


9 


3 


5 


4.7 


.5 


9 


4 


5 


4.6 


1.0 


9 


2 


5 


4.7 


.7 


9 


3 


5 


5.0 


0 


9 


5 


5 


5.0 


0 


9 


5 


5 


5.0 


0 


9 
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• There should be more discussion of locating appropriate comparison 
groups, especially for classes such as kindergarten or special ed. 

• Regression should be explained better. Avoiding regression bias 
requires too much testing. Wiat is spuriously inflated? How much? 

• It is not possible to administer early posttests to students when you do 
not know they are leaving. 

« How long should a student participate in the project over the year to be 
considered a participant? 

• Discussion on missing data is misleading. 

« Explain the differences and similarities between the Gap-Reduction 
Design and the Norm-Referenced Design. 



Chapter VII. Co llecting Outcome Data 

• The expectations in this chapter are unrealistic for a large, urban pro- 
ject. 



Evaluator Coimnents Chapter? VI and VIT^ 

The two chapters of the Users' guide that deal with implementing the 
gap-reduction design (Chapter VI) and processing and analyzing data (Chap- 
ter VIII) were discussed separately with conference attendees who consid- 
ered themselves to be evaluators and those who considered themselves to be 
project directors/administrators. What follows are the comments and sug- 
gestions of the evaluators. • 
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Ph^ptey VI, — Iinplementtng the Gap- Reduction De fsfRn 



None of the evaltjators had any problems with the content of the chap- 
ter, although there was considerable discussion of the RGI metric. Ear- 
lier in the conference it had been noted that RGis become meaningless if 
the comparison group makes zero growth or negative growth. While this 
situation should never arise with achievement test scores (i.nless they are 
expressed in NCEs — a conversion that reduces norm-group growth to 
exactly zero), it could arise with non-test data such as attendance and 
with other measures including classroom grades where growth is not 
expected. 

A discussion of these issues is not currently in the Users' Guidy ., 
and the consensus was that it should be. It was also suggested that the 
purpose of the BGI metric (to provide a comparable "ecore" useful for mak- 
ing comparisons between different projects using different tests and dif- 
ferent comparison groups) be explained. It should also be explained that 
Where a metric d9es not have to perform equating functions, evaluators 
need not compute RGIs or even standardize growth estimates. Under these 
conditio. ;s, the amount of gap-reduction itself provides all of the needed 
information. 

Keith Baker suggested a refinement of the gap-reduction measure for • 
possible inclusion in the BEES. This refinement would express the amount 
of gap reduction as a percentage of the pretest gap. It would be particu- 
larly informative if the mainstream classmates of the project participants 
were used as the comparison group. A lOOZ gap reduction would then mean 
that no gap remained and that the intervention's job was complete. 

Following the technical discussions just described, we worked through 
some hypothetical examples of how the gap-reduction design could be used 
with non-test data. 
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. Ph^Pter VIII. Procgssing and Analvzlnp Data 

The evaluators reviewed chapter VIII near the end of the conference 
under considerable time pressure. Ve did not get past p. VIII-6. 

The first issue addressed was score conversions (p. VIII-2). The 
psers' Guide presents this as a necessary step in the data processing 
procedure. Elsewhere, however, it says that raw scores may be used. The 
Vs^ys' quid? should make clear when conversions are required and when they 
are not. Why are raw scores less desirable? In general, a discussion in 
the a^ex^' G^ide is needed to explain the different types of scores and 
how they should be used for analysis and reporting. 

The construction and management of data bases was another concern. 
It was felt that not enough guidance is presented on this topic. It would 
be nice if the V^eys' Qu%de could present information about different com- 
puter software— what will work on what computer, what each package can- and 
cannot dc, and how much each costs. It would also be nice to know how 
difficult/easy each package is to vise. Most people will need more spe- 
cific guidance than is presented. A flow diagram would also help. 

It was suggested that we give more emphasis to recording student ID 
numbers aad grade levels on all data collection instruments and files. 

One evaluator felt that classroom grades were too unreliable to vise 
for monitoring the progress of reclassified LEP's. He described classroom 
grades as "just one option and a bad one at that." He felt we should rec- 
OBunend the use of professionally-developed tests whenever possible. He 
also felt that ratings or rankings made by classroom teachers would be as 
unreliable as classroom grades. 

We were just beginning to discuss Other Sximmative Outcome Analyses 
when time was called. It was clear, however, that the Users' Guide dis- 
cussion of baseline data was confusing to some and needs work. 
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IV. LESSONS LEARNED FROM THE FIELD TEST 



The lessons learned from the field can be roughly categorized into 
two groups, those that pertain to the system itself and those that pertain 
to the ysers' Guide. Each of these categories is addressed below. 

Lessons Related to the BEES 

In general, the sites' reactions to the BEES were positive. Both 
during the course of our site visits and during the first day of the Exit 
Conference, there were many comments relating to the evaluability assur- 
ance component of the system. Between the Users' GuldR and the technical 
assistance provided during site visits, most of the field test partici- 
pants felt that their planning and evaluation activities had been signifi- 
cantly improved. BEES recommendations helped them to state their project 
objectives more clearly, to design project activit: . so that they related 
more directly to the objectives, and to focus their evaluation questions 
more sharply. " 

All sites found the recommended docmentation and supplementary ana- 
lyses excessively burdensome. They would like more direct guidance as to 
which of our recommendations are "driven" by the regulations and which are 
not. Among those not tied directly to the regulations, they would like to 
see some prioritization. • ' 

Most of the sites had on6 or more problems with our recommendations 
regarding test selection. Some have no choice in the matter and have to 
live with the state- or district-mandated instrument, others are simi- 
larly precluded from functional -level testing, from modifying test 
instructions or time limits, or from making any Changes whatsoever to the 
test items. Even when our recommendations could be implemented, some 
(e.g., reviewing candidate instruments item-by- item) were considered too 
constuning of time and effort. 



57 

63 



The sites vers also very positive about the gap- redact ion approach to 
assessing project impact. Several - -:^cors expressed the opinion that 
the deaign was the only one that made sense and was feasible to implement. 
On the negative side, several districts found the required computations 
frighteningly complex. They would like us to provide user- friendly soft- 
warv^ that would free them of all the computational burdens (including 
interpolation/extrapolation and correcting for regression). 

D£t;a base management is an aroa where all but one site felt that the 
BEES' recommendations are inadequate. If nothing more, we should describe 
existing software that they might be able to use. 

Project directors and teachers believe that classroom observation is 
very important to use in program evaluation. It seems likely that these 
staff members will conduct most, if not all of the observations. 

Except for information regarding comsunity characteristics, most of 
the data the BEES recommends collecting for evaluation and reporting 
appear to be accessible and potentially viseful to project staff. 

The BEES does not have enough emphasis on evaluating parent involve- 
ment; staff training, and materials development. 

To supplement the preceding, somewhat subjective impressions,' seven 
general questions regarding the BEES were posed to each of the nine field- 
test sites on the final day of the Exit Conference. These questions and 
the written responses of site personnel are presented below 

OMestton 1, To what extent has t h e system aided the loral prolActs -In 
def^Tlln R an4 pbieptiveXv measurinK the impact n f the bilitipnal progr»in on 
its students? 

Four sites responded that the BEES had been helpful by providing a system 
in which the relationship between needs, objectives, data collection, and 
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monitoring of effectiveness is clearly defined* Two sites said that it 
was too early to tell at their sites, two sites were vaguely positive ^ and 
one site said that the gap-reduction model should be useful • 

Puest^op 2, — yo_what extent can the system serve to improve local evalua^ 
tlon practi,^.es7 

Two districts responded that the BEES had helped them make use of district 
evaluation data and other existing data in their evaltiations • Three dis- 
tricts gave general but positive responses, and two said the system would 
helpful only to the extent that funds are available to implement it. One 
district said the BEES could improve all local evaluations involving 
minority students, and one said the system had been helpful in encouraging 
the project director to participate in the evaluation. 

Quest;tpn 3t — To. what extent ca n the system gener&te data suitable for use 
In local decision-making? 

Two districts said that the information would be useful in gaining school 
board understanding, while a third district said the information would be 
useful at the project level but not at the district level. One district 
said the data generated would depend on che availability of resources to 
implement the system, and three districts gave general but positive 
responses « 



.Qt^est:ion 4, — How adequately can the evaluation destfms be integrated with 
ft district regular instructi onal and testing pro frramg? 

Pour districts said the evaluation design might or would be very difficult 
to integrate with district practices. Three districts said it would fit 
in quite well, and two said it was too early to say. 
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Question 5, — How suitable are the data j^et fie rated bv the system for svnthft 
sis acros s Title Vtl^programs? 

Three districts said outcomes should not be combined or compared across 
different types of projects and populations. Three said results would be 
suitable, and two said they would be suitable only if the RGI is actually 
workable. One district said they had been hoping for much more emphasis 
on data integration and synthesis, including software. 



Question 6, Wtiat pyp. the costs Incurred In i iimleinentfr.f> t h'' oTr«],.nf-4^^ 
designs described in the Users' Guide ? 

• Costs associated specifically with the project site (e.g., specific to 
setting up and maintaining the records that the system requires) ; 

• Start-up costs (e.g., training data collectors); and 

• Ongoing costs of evaluation (e.g., collecting and analyzing data that 
are collected) . 

Two districts estimated costs in dollars. One estimated site costs of 
$30,000, start-up costs of $5-7,000, and ongoing costs of $6-7,000. A 
second district estimated $1,000 site costs, $1,000 start-UT) costs, and 
$8,000 ongoing costs. Five districts said that costs would be high, vith 
three adding that at least one full-time person would be required. Two 
districts did not know. 



Question 7. To what extent would a district require technical asststanrp 
In imple inentln^ the sy g^Piny 

Two districts said a full-time evaluator would be more practical than 
technical assistance. Six districts indicated that they would need help 
in data management and analysis, possibly through the provision of appro- 
priate software. One district said a little technical assistance would be 
necessary but did not specify the area of the assistance. 
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Ussons Related to t he Users' fltildp 



The Pggrs' guide has already tindergone several major revisions. 
Clearly, however, substantial room for improvement remains. Perhaps its 
most significant shortcoming is its failijre to communicate effectively 
with the disparate audiences. Its more technical parts are viewed as 
incomprehensible and full of jargon by program people. Evaluators, on the 
other hand, find the non- technical parts irrelevant to their task and the 
technical parts excessively discursive. Both technical and non- technical 
readers perceive the writing as a compromise that fails to serve either 
group as well as it might. 

Throughout the fiuids, there is a consensus that the writing style and 
the format could be improved. There are inconsistencies of style, head- 
ings, structure, and punctuation that need to be fixed. With somewh.-it 
less consistency, field test personnel and other reviewers made the fol- 
lowing suggestions: 

• the Users' Gujde should include a glossary of terms. 

■I 

•V V ^J^^^^^^^^^^^ should be stated more directly 

• more rules of thumb should be given 

• more step-by- step procedures should be given 

• there should be more examples 
e there should be more explanation 

• the document should, be shorter 

• there should be less explanation 




ERIC 



61 

67 



• there should be separate Guides for project directors and evalu- 
ators 

We have even received such radical reconnnendations as, "limit the 
Vg^rg' Gut^? to the gap-reduction design and discard everything else." One 
reviewer suggested just the opposite: "throw out all the outcome evalua- 
tion material and limit the Guide to process evaluation. 



Conclusions 

It will obviously be impossible to incorporate all of the recommend- 
ations into a revised BEES and Users' Guide as some of these are mutually 
contradictory. Many changes will be made, however, to make the system and 
the Hssral_Guid£ better meet the needs of their audiences. In approximate 
decreasing order of priority, we plan to do the following: 



• provide software that will perform all of the gap -reduction compu- 
tations 



1^ 



0 . 



^ 9 develop an abbreviated Users' Guide that will present rftcommend- 
ati5>ns succinctly and without any unnecessary explanations 

^ • present parallel conceptual (for project director) and technical 
^ (for evaluators) segments of the Users' Guide whenever psychomet- 

ric and statistical issues must be addressed. 



develop some recommendations for data-base -management software 

separately identify required, important but not required, and 
probably useful categories of data to be collected and analyses to 
be done 



• include a glossary of terms (already nearly completed) 
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« jBake our recosamendations more directive and procedural 



• include ipore examples and rules of thumb 

« 

• vindertake a major editing and reformatting revision of the Users' 
Guide iising professional document-design specialists. 

It is not yet clear how we will respond to comments on the test 
selection/modification issue. Clearly we need to alter what we have, but 
we need to avoid conveying the impression th>.t any old test will do. Per- 
haps we have been overly zealous in our recommendations regarding func- 
tional-level testing and modifying existing instruments. On the other 
hand, users of the BEES must be made aware of the fact that test floor • 
effects and low curricular reliability can only make their projects appear 
less effective than they really are and that other measures of effective- 
ness must be sought if state or district mandates force either of these 
ill-fated circumstances upon them. 
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There is one, overriding criterion for the selection of field test 
sites of the "Bilingual Models" study. That criterion relates to the need 
to try out all parts of the User's Guide and associated materials. 

Most parts of the Users' Guide will be applicable to all bilingual 
eduction project evaluations and thus will have no particular implications 
for field-test site selection. One very important exception to this gen- 
eral rule, however, is that portion of the Guide that deals with dividing 
observed growth into treatment-related and non- treatment-related compo- 
nents. Three alternative designs will be offered as means of accomplish- 
ing that objective (quasi time series, regression-discontinuity, and non- 
equivalent comparison group). We anticipate that many projects will find 
it impossible to implement any of these designs. Since they will be part 
of the Users' Guide , however, it will be our highest site-selection 
priority to find at least one field-test site that can and would be wiling 
to tryout our procedures for implementing each of these designs. It would 
be desirable to find two or three sites that would implement each design. 

Our second-priority site -selection criterion will be diversity in the 
extent of projects' instructional usage of LI. Because different types of 
tests and testing procedures will be required, we would like to include 
projects that encompass LI literacy as an academic subject, projects that • 
use LI as the medium of instruction for at least part of the curriculum 
(but do not teach LI as a subject), and projects that use English as the 
medium for all instruction. 

Our third-priority criterion relates to the linguistic characteris- 
tics of the students served, as different characteristics impose different 
requirements on both instructional and evaluative processes. We would 
like to include projects serving Hispanic students, projects serving Asian 
students, and projects serving students with heterogeneous language back- 
grounds. Our preference would be to include at least one project serving 
Hispanic students o£ Mexican origin, at least one project serving Puerto 
Rican students, and at least one project serving Cuban students. Among 
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projectii serving Asian students, we would like to include at least one 
serving refugee students and one serving non-refugees as we again expect 
that the two groups would pose different problems for both educators and 
evaluators • 

We would el'5o like our sample to include projects of different sizes, 
projects serving different grade levels, projects whose evaluators have 
different levels of technical sophistication, and projects from different 
parts of the country. With only nine districts in our sample, of course, 
we may not be able to achieve all of these objectives. It will help, how- 
ever,- if we can find districts that have more than one project. 

Finding districts for the sample and getting them to participate will 
probably bcth be non- trivial tasks. Our tentative plan is to begin by 
contacting the bilingual program directors in states with large numbers of 
programs. Since California, New York, and Texas have something on the 
order of 85Z-of all the programs in the country, we propose to begin with 
them. After explaining our objectives for the field- test seonple, we will 
seek nominations. While following up on the nominations received from 
these states we will also seek nominations from other states, from the 
Evaltiaticn Assistance Centers, and possibly from OBEMIA. 

We will attempt to obtain a large ntimber of nominations, both to 
allow for sites that will elect not to participate and to provide us with 
flexibility in filling our sampling frame. Before we begin contacting 
sites we will attempt to determine the characteristics each of them pos- 
sess vis-a-vis our selection criteria. We will then begin soliciting 
their participation starting with 'jhe sites that have the most desirable 
characteristics . 
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The characteristics of the first site that agrees to participate will 
cause us to adjust the order in which subsequent sites are contacted. 
Similar reorderings will occur as each successive site is added to the 
sample. This approach, although cumbersome, is the one we believe will 
produce the best possible sample. 
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APPENDIX B 

Protocol Invitation f or Fleirt J ^ st Part^c^p »^f 



B-l 



Dear Superintendent: 



This letter is a foraal invitation tt> the 



school district to participate, as one of nine school districts across the 
country, in a field test of the bilingual education evaluation system we 
have been developing under a contract with the U.S. Department of Educa- 
tion (see attached project summary) . As I have discxissed briefly with 
you, and more extensively with members of your staff, the new system is 
being developed in response to the widely recognized need to improve and 
standardize procedures for evaluating bilingual education interventions. 

The ultimate goal of the system is to enhance the quality and effec- 
tiveness of the services being provided to limited-English-proficient stu- 
dents. For this reason, it has a heavy emphasis on process evaluation and 
on the integration of process and outcome information for the purpose of 
generating recommendations aimed at improving local projects. The pur- 
poses of the field te?t are to assess the feasibility of implementing the 
system as we have developed it and to identify changes that could be made 
to enhance its ease of implementation and to increase the utility of its 
information return. 

Agreeing to participate in the field test implies that your staff ' 
will make a conscientious effort to implement our recommended evaluation 
practices to whatever extent situational arid budgetary constraints will 
allow. We will not be gopducting the evaluation in your district, only 
assisting responsible persons in the district to try out the procedures 
and materials we have developed. 

Participation in the field test will represent some burden for your 
district. We believe, however, chat this burden will be more than offset 
by the benefits that will accrue to your district through participating. 
The evaluation expertise of your staff will be substantially enhanc^ed by 
virtue of the training and technical assistance we will provide. Further- 
more, by the end of the year, you will have established sound evaluation 
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practices that will stand you in good stead over future years. During the 
fi^d-test year itself we will also be able to assist with data analysis 
and interpretation and perhap.s provide you with computer software that 
will also be useful in the future. The greatest benefit of all, of 
course, will be program improvement if the new system does indeed r-ove 
useful for that purpose. 

As previously planned, your involvement in our filed test will 
involve sending your Title VII project director and one or two other per- 
sons associated with bilingual program evaluation to our office for one 
week of trainings We will pay their travel and related expenses during 
that we< k, but cannot pay salaries or honoraria^ The training will occur 
not long before, or shortly after the school year begins, depending on the 
availability of the parsons to be trained. 

During the school year, members of our staff will visit your project 
probably six times before and/or during critical evaluation activities. 
At" those times we will monitor evaltiation activities and provide technical 
assistan-,e. Our purpose will not be to evaluate the manner in which your 
evaluation activities are being conducted but to identify deficiencies in 
the procedures and materials we have developed so that they can be 
improved. At the end of the year we will conduct a debriefing. 

During the course of all our visits we will need to have access to 
your project director and evaluator(s) as we will need to discuss the ade- 
quacy and xisefulness of our materials with thcira. We will attempt to be 
unobtrusive, however, and to conduct our discussions on a non- interference 
basis. We will, of course, treat all daU\ provided by your district as 
confidential and will take whatever steps are necessary to protect the 
source of all information we collect* 
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As we are eager to begin by sending materials to your staff for 
review, I hope that, you will let me know your decision as soon as pos- 
sible. Should you have any questions, I will, of course, be pleased to 
try to answer them* 



Sincerely yours, 



G* Kasten Tallmadge 
Senior Vice President 
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APPENDIX C 

Agenda? ^qd Corr^spondRnre R6lflt:^r,fr t » the Two Project Work^l^ 



9SS. 



EMC 



C-l 




SRATechnoIofli8S.lnc. 2S70 Wesi El Camino Real Suite 610 Mounlain View. CA WWO 4,5.94,.70M 



27 October 1986 



Norma Anzaldua 

Valley View Independent School District 
Rt. 1, Box 122 
Pharr, TX 78577 

Dear Norma: 



During the last day of our workshop, I believe I promised to . > 

After my fim phone xi^tl to you and t^j^.^bthers. I felt I had better 
keep ny prool'se after all. - - ^ m 4. better 

'■3' - :t<15& ' V- •:•■«■'■ ■:•¥• 



V --XT'. 



In general, what I have In Bind for you to do is to have vou und 
your evaluator (and other Involved individuals) read over the U^rs" 
to detemine what recommendations can be implemented a^^gg- 
recommendations cannot, and wh^. The next step is to aake planT to 

Z^'"' '«=-«°dations. Af?er tLr^pl^^t the 

identified recommendations and keep records of progress. S^role is 
act as facilitators and to document all these activities. 

Let ne highlight some of the key things that you nay want to 
consider doing at this point in time. Th^ are: 

1. Review your proposal and evaluation report with your evaluator 
and relevant staff for the purpose of clarifying and modifying 
(if needed) your needs statements, project objectives, and proj- 
tlLliT. ^l^^T^"^ guidelines presented in the EvIluatiSn 
Focusing chapter. Com^ up with some evaluation questions if you 
don t have any, or modxfy your exlstip- evaluation questions if 
they do not meet the criteria listed .a the Focusing chapter. 

2. Together with the relevant "othera." deteralne your project's 
global and specific critical program features which you can 
observe and/or document Via other procedures such as question- 
naires or interviews, plan all these activities now. 
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example, involve aore people in the evaluation process con! 

KvTL'^h^ork^'^:s/ t:o°'sXie; i 

DiSalL !J y ? * anticipate potential obstacles and • 
ioplenent some of the strategies for increasing 
the utilization of evaluation findings. "creasing 

"!!^ projects, perhaps you aay have the luxury of select- 
JSf;.!J or developing your «B;e««.ent ini??SSn?;. 1?' 

u^l. ?°^u" recoaaendation. as your guide- 

lines. For the continuation projects, it will bTusefS to 
review the quality of the instrument. th..f you a^e "SS Sth 
using the test selection criteria described iS S^aptS 4 ^1 
Bay want to consider modifying your test as suggested in ChSter 
4. You aay want to develop instnmeats for asS^ proerS 
i«.pleae^..tion. Read Chapter 4. Is it usef"? Ir^thl S^- 
Kendations; do-able? m Chapter 4. the reco«aendationJ not t^ 

^' vorkshop Day One exercise' (attached)? That awful 

list of variables that you were asked to look ovor and ratf 
Make plans to collect them now. I'd like to k^«r wSy yof L't 

. l^To^Z; It^Tus^ • - 

implementing program and evaluation activities (see Chapter 3, 
: u il r' teachers, and (perhaps) the eval- 

uator should keep a log of this type of informatJon. which 
ahould be consolidated periodically (e.g.. every stiff feting) . 

^* iLlll "^l ^""^ '° "^^^"^ eoDstructing your 

data management system. Consult Chapter 5 (V-22). 

8. Hake plaM tb' collect data following recommended procedures 
presented in Chapter 5 (V-1. V-15)7 proceaures 

i« """^ °^ that come across my mind. There 

ySu? 5Hi^ carefully and cLnfup StJ 

IZliT^ ^ • ^! flHt-visit will be to help you to 

finalize your agenda of things to do for the field test? thiLs you can 
do to improve your evaluation and your oroeran Pro™ ^^mo !4 
be^se^dlng ,c„ part, oi the ST^n „'p3:t.?:r're5J:ti Ir" 
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As I Mentioned to you ever the phone, bost likely we will have 
another get together" Booetlie In late June or early July. At that 

and whjr. After Individual presentations, we" will then discuss as a 
group or subgroups the merits of the evaluation system (BEES) followlnff 
lu'' ir^^ 5"^'!^?*^ ^^"^^ t*'^ workshop handout labeled, "Purpose of 
the Field Test" (attacaed) and the content of the Users' Guide foUowL 
the six points listed in another handout labeled "Open Discu Mlon' 
Content of the Users' Guide" (attached). Throughout the year when 
you're working with the Guide , it will be useful to keep in tnind the 
above issues that are the foci of the field test. 

^.^ 

K - ^""^ gathered by now, we're really asking a loti Yes. 

but that's because we really value your input which we believe can 
significantly improve the quality of the Guide . After all, you are the 

n2r!?ii°^ ^i"''^*'^* proJcItTIud most importantly, your 

jn^ will be remembered forever in the annals of bilingual education 
evaluationl Just think, this clout, plus your enhanced skills In 
proposal writing and evaluation will guarantee endless awards on your 
future proposals. uu jrour 

■Talking about proposals, I think it will be a great idea if you 
would use- the Guide to help you write your next proposal. Not only will 
that create an opportunity for you to study the Guide, I also believe 
that the quality of your proposal should be enhaSced"(that aeans a 
better chance in winning the big bucks I) 

Jokes aside, »e really appreciate your help. I believe if we all 
give it cjr best shot, the field testing will be .a rewarding experience 
for you and us. I'll make sure the benefits are mutual and tS^Sork S 
as omch fun as possible. 

Talk to you soon. 

Sincerely » 




Tony C.H. Laa 
Project Director 



TL:TH 
Enclosure 

cc: Ray Morales 
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Cheryl Crawley 
Salom-Keizer Public Schools 
1309 Ferry Street SE 
P.O. Box 12024 
SaleD, OK 97309 

Dear Cheryl; 



,...0. J^^^.fJ^f*^ ''^^-^"S f""' before you know It. the June 

o Evaluation System (BECS) will be history. I feel 

Jo ir J^*; f'^"'* "-^^^ '^'^ '° """Sest to you thloK« yo7^y want 
to wrap up fron now until June In preparation for your prSeutatlon in 
the upcomxng ccetinK in Washington, DC. presentation in 

To reiterate what we've discussed in the past, your role Is to 
provide us with feedback fron ycir group's perspecave as pract!- 

«trengths and weaknesses Wt the BEES' gse"' Gu ide. 

c^efunv' n'rni? ."V' '"'f""^ '''' Users' C^ide " 

(prere^LrlJ if 1 J' ""'™«"''«^ions in two wa ys. The first 
r^irJtfr ^ ""'"^^ lnipleme7-.tation of the recomendations. e.«.. 
rewriting the project objectives. The second way is to revi^the 
recoumen^tions that you are „m»ble to implement. Then Jou S^oulS 

ora::S 1"*^°"^"'*^'°"^ i^nplcn^nt^bU^InS'llch 



There are four things I'd like you to try out. Thev are- l»\ 
•revise project objectives, (h) develop evaWiSI questionrio 

de "JtL'i'SLJ^friT'^""' 2uidc Unwinding 

descriptiv*. information regarding problems associat?d^ith prof ram 
inpleoentation and evaluation activities), and (d) revJew eSstS 
test inetruuants (at least for one grade for one subtest" c^j!. 
Refldlng Comprehension). * ^ * 

1 I enclosed a copy of the data collection survey with this 
J:ii:;ienJf '^^'^'^^^'^ P^"- - at'ySr earliest 
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" I m not m the office, leave a ceaeage and I'll get bacK to you. 

to H^?""" ''^^ ^^'^^^'^ ° consultant who can asaist you in vecard 

-t tt .Tn'T'''''* J" ^'"d" «nd Ghe^^n be readied 

-t the sane phone nucber as nine. She'll be lu touch with yo^ aJon. 

I know U|s a" lot of work but 1 believe the rewardt; should b« 
Just as plentyl Take care and I look forward to seeing y^u^gfi'! 

Sincerely, 



Tpny Lan ' . . . 
, . I *:roJect. Director 
TL:TH * • . . 

Enclosure 



i 



C-6 C2 



Kvaluatlon Assistanee renter-Went 
Data ColleetjQa and Ri>pn.^f tng Survey 

a lor Aitie VH local project evaluations. 
Please rate each of the followine Hat of voWoki . 

acceaalblUty in your district. variables in terms of availability and 



STUDEWT IHPORHATTnM 

A. Backg-rounrf Informatlnn 

Date ot birth 

Place of birth 

Ethnicity/ language groii.j 

Date began U.S. residence (date) 

Language used in home 

Fluency in Engliflh as of (date) 

Fluency in native language (LI) 
if other than English as of (date) 

Years of education completed as of (date) 

Years of education missed as of (date) 

Family income status (e.g.. AFDC 
participation in National School* 
Lunch Program) 

Public or private school 
School grade level 
Be Seirviee Data 

Dates entered and exited project 

Other instructional settings (e.g.. 
umlnatream class, Chapter 1 progfai, 
special education, migrant program, 
KSL program) 

Dates entered and exited other settings 



Information 
Available? 

Bon't 
Yes No Know 



If Not Available, 
How Accessible? 



Very 
Difficult 



Y 
Y 



N 
N 

N 



DK 
DK 

DK 



Y N DK 

Y N DK 



Very 
Easy 



Y 


N 


DK 


1 2 3 4 5 


Y 


H 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


12 3 4 5 


Y 


N 


DK. 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 

12-345 
12 3 4 5- 
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Information 
Available? 

Don't 

iSS. No Knov 



Y N DK 

Y N DK 



C. Hon-Test Data 

Years retained In grade as of (date) 
Whether or not student dropped out 
Day« absent from project classroom as 
af oflhat^'te'^'^' '^'^ 

Date of special education referral 
or placement 

Date of placement in gifted and 
talented progra' 

Enrollment in postsecondary 
education institution 

Setting from which entered project 

Grade point average prior to entry 
in project ' 

Setting to which exited from project 
Grade point average after exit 
D. Former Proj ect Studenta 
Standardized achievement test data 
Classroom performances (e.g., grade) 

II- IHSTRUCTIOW 

-Academic and non-academic subjects 
taught in the project 

Method of service provision (e.g.. 

tutoring, classroom Instruction) Y N DK 

The following variables apply to each class or subject matter offered 

Y N 



If Not Available, 
How Accessible? 

Verj- Very- 
Difficult Easy 



1 2 3 4 5 
1 2 3 4 5 



Y 


N 




1 2 3 4 5 


Y 


N 




1 2 3 4 5 


Y 


N 




1 2 3 4 5 


Y 


N 


OK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


1 2 3 4 5 


Y 


N 


DK 


12 3 4 5 



English and LI materials and how they 
are used ' 



Y N 



DK 
DK 



1 2 3 4 5 
project: 

1 2 3 4 5 

1 2 3 4 5 



ma 



Infonaatlon 
Available? 

Don't 

Pettern of claBsroom language use Ce ^ — ~ 
concurrent, Previe./revlfv.'or^^Le^fjing) y 



newcomer UnL%ni\'rSoS""°- 
students are together all ^y)' 

aJuSt ^^uL^L^''"-"-! 
abilities) -^nsuage, language 

Function of aides in classroom 
(X instruction, Z nmnagement) 



m. STAFF 



Academic preparation of each staff 
«nenib?.r, Indudjlng: 

• level of education, 

• eXlttnr^""'^" 

e credentials and certificates, and 

• academic preparation in field of 
biangual/bicuitural education 



N DK 



■Y N DK 



N OK 



Y N DK 



N DK 



N DK 



Y N DK 



Y N DK 



N 



DK 



Y n" DK 



Y N 

Y N 



DK 
DK 



If Not Available, 
How Accessible? 

Very 

SimS^lt Easy 
1 2 3 4 5 



1 2 3 4 5 

1 2 3 4 5 

1 .2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 



Y N DK 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
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Teaching experience of each staff neaber. 
including: ■ 

• years of experience in laonollgual 

• classroomBj and 

• in bilingual classrocos 

language abiUties of each staff menber, 
including languages understood and 
spoken and degree of fluency in: 

• English 

e LI 



Please answer the following questions: 



Infommtlon 
Available? 

Don't 
Yes No Knou 



If Not Available, 
How Accessible? 



Very 
Difficult 



Y 

Y 



N 
N 



DK 
DK 



Y 
Y 

Date: 



N 
N 



DK 
DK 



1. What is your position? 



2. Are you Involved in: data ollection? Yes^ 

data analysis? Yes 
reporting? Yes 

3. How many students are currently enrolled in your Title VII project?^ 

4. What grade levels does your project serve? 



. No. 
.No_ 
No 



5. What year is your project In? Ist 2nd 3rd 4th_ Sth- 

6. How many students (approximately) are enrolled la your distrlct?__ 

7. Are your project data stored in computer? Yes No 

8. How are your project data analyzed? by hand? Yes 

■ by computer? Yes_ 
both? Yes 

9. In your district, do you have a tet ting/evaluation office which: (check) " 

has a centralized computer system? 
provides you with some student information? 
assists you in your data management? 
assisfs you in your testing? 

10. (Optional) What is your district?. 



Yes. 
Yes_ 
Yes. 
Yes 



No^ 
Ho 



, No^ 
NO 



Very 
Eas^ 



1.2 3 4 5 
•1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 
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RMC RESEARCH CORIORATiON 
2570 WEST EL CAMiNO REA:. 
MOUNTAIN VIEW* CAUFOflNIA 
TELEPHONE* (415) 941-8550 



May 19, 1987 



Sarah Clayton 

Director of Categorical Programs 
Valley Center Union School District 
28751 Cole Grade Road 
Valley Center, CA 92082 

Dear Sarah: 

reporSnf liivey' ^T^if iH Ln"'^ ^ * '^'^ collection and 

teporcang survey. Thig is a follow-up letter, which I hope will orovide von 
with scm guidance in preparing for the June raeeting. Provide you 

site. The contents of your presentation should include the following: 

overview of your project (title, grade levels. 
lanBuage(s}, staff, academic or nonacadeaic subjects offered 
nuniber of schools, number of students) . "rrerea, 

WeiL?" "°;"°«"«^tions implemented, activities undertaken to 
implement each (e.g., meetings dassroPm visitations, ar»d so on) 

o A brief description of the products resulting from implemenfine 
the recommendations (e.g., revised objective!, evalSon ^ ^ 
questions, classroom observation form, RGI, and so on) 

SSemStef ^"ociated with each recommendation 

On the second day, we will go over the User's Gui.de, chanter bv chao-or 
as a group, at which time I expect each site to provide is S feedback on 
the content (recommendations. ex«:pies. table.) , format and vritine fL . 
recon^endations that you have ncr implemented, we J^l fjke^o & ly 

woui?me to have ST"? ^^^^^ ^Iso, S yZ' 

wouio like to have us include tha-. is xiot in the guide. It is Iraoortant ttiat 

S?e"1 We'iS'S r? "? that repp's s y^ir'"'' 

elb'e'ddeJ Jrthe"S:' '^'^ recommendations ^t are 
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Sarah Cl&yton 
May 19, 1987 
Page Two 



I'll call you after you have received this letter. 

Sincerely, 




— r 

Tony Lm 

TL:mb ^^^i' Director 

Enclosure 
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" "arcl, 1987 



Cheryl Crawley 
«aien.-Keleer Public School. 

Hrry Street lit 
f.u. box 12024 
iJalec., UK ■y730y 

Ue«r Cheryl: 



"nt win be In WaKhlnRton br 

expe,.s«. would li J v^. T '^"'""^ (including yoursMf ^ . o . 
^'^'i ^ ready ..olo by .^'Ju a ''r'^' -^^^^^^^^11^' ' 

noon on Friday the lyf,,. °" ^''^ ^^^h. oxpact to Ireak Jp^y 

°e"=tl«g/'^^hiris''Ju^rar"^' t-^'*''*'' ^in^^-ed the a.end. f u 
you aftain as Knn„ - ^^'^^J* «arnJr.K' noLlf<,.«.^ ^Kenda for the 

obouf trav^ ° ^'^ ^'^^^ ^^^e details 1„ "^^1 contabt 

tickets for your group by mio-HBy. '^^ervatlons and buy 

i'lease call we if you 

y°" have any questions. 



Cordially, 



^' '^«««-en TallnadRe 



BILINGUAL EDUCATION EVALUATION SYSTEM (BEES) 

KOBKSK)P iftSEHDi 

San Francisco Airport Marriott Hotel 
October 7-9, 1986 



Titeadtiy. October 7 

Topic 
A* Introduction 

© Welcome and self introduction 

d Project background 

• Purpose and g' ldelines of 
field test 

a Workshop goals and agenda/ 
participant expectations 

9 Pretest 



BREAK 



• Overview of BEES 

B. Evaluation Focusing 
Evaluation Planning 



Presenter 

Kast 
Kast 
Tony 

Tony 

Staff 

Tony 

Tony 
Tony 



Time 

8:30 - 9:00 
9:00 - 9:15 

9:15 - 9:30 

9:30 - 9:45 
9:45 - 10:00 
10:00 - 10:15 

10:15 - 11:15 
11:15 - 11:45 



L U N n H 



11:45 - 1:00 



D. Process Evaluation 

• ./hat data to collect? Tony 

© What instrument to use? Tony 

© How to prepare for data collection? Tony 



BREAK 



• An example 
E» Review and questions 
F. Posttest 



Hsctor 

Tony 

Staff 



1:00 - 2:00 
2:00 - 3:00 
3:00 - 3:15 

2:15 - 3;30 

3:30 - 4:35 

4:35 - 4:50 

4:50 - 5:00 
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Wednesday. October 8 (Presenter : Kas t) 

Topic 
A* Pretest 

B* Basic Strategies for Outcome Evaluation 
C. Measurement Error and the Regression Effect 

BREAK 
D» Selecting an Achievement Test 

E. Other Achievement-Related Indicators 



LUNCH 



F* Test Administration and Scoring 

Calculating Growth Indices 
He Review and questions 

BREAK 
I. Quasi-Experimental Designs 
J* Posttest 



Time 




9:00 


- 9:10' 


9:iO - 10:00 


10:15 


- 10:30 


10:30 


- 10:45 


10:45 


- 11:15 


11:15 


- 12:00 


12:00 




1:15 - 


2:00 


2:00 - 


2:45 


2:45 - 


3:15 


3:15 - 


3:30 


3:30 - 


4:45 


4:/>5 - 


5:00 
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Thnrsday. October 9 



Topic 

A. Data Processing and Analysis 

B. Integration and Interpretation 
of Results 



Presenter 
Kast/Tony 



BREAK 



C. Evaluation Reporting 



Kast/Tony 



Tine 

9:00 - 10:30 

10:30 - 0:45 
10:45 - 11:45 



LUNCH 



11:A5 - 1:00 



D. Overall review and quepfcions Kast/Tony 

E. Open Discussion: Users* Guide ^11 

BREAK 

F. More Open Discussion: Field Test All 
6. Wrapup • 

H. Vorkshop Evaluation 

I. Individual Questions/Farewell 



1:00 - 2:00 
2:00 - 3:15 
3:15 - 3:30 
3:30 - 4:00 
4:00 - 4:15 
4:15 - 4:25 
4:25 - 5:00 
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AfiEHM FOR HBE7IN6 



Washington, DC June 17-19. 1987 



Wednesday, June 17 



Topic 



A. Introduction 

- Welcome 

- Overview of Field Testing 

- Overview of Meeting Agenda 

B. Field Site Staff Presentation 

- Salem, OR 

- Polk County, FL 
BREAK 

- Tucson, AZ 

- San Jose, CA 
LUNCH 

- Valley View, TX 

- Valley Center; CA 
BREAK 

- Community School District #10, NY 

- Community School District #5, NY 
BREAK 

- Tyler, TX 



Presenter Tine 



Kast/Jim 

Tony 

Tony 



8:30 
9:00 
9:15 

9:30 
10:00 

11:00 
11:30 

1:30 
2:00 

2:45 
3:15 



9:00 
9:15 
9:30 

10:00 
10:30 

11:30 
12:00 

2:00 
2:30 

3:15 
3:45 



4:00 - 4:30 



Thursday, June 18 

C, Group Discussion: Review Contents of Users* Guide (Vol. 1) 

- Introduction Tony 8:30 - 8:4. 

- Chapters 1, 2, 3 
BREAK 

- Chaptfcr 4, 5 
LUNCH 

- Chapters 6, 7 
ilREAK 

TChapters 8, 9, 10 3:15 _ 4.45 

•-Overall Evaluation of the BEES 4:45 - 5:15 



8:45 - 10:15 
10:30 - 125100 
1:30 - 3:00 
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Friday, June 19 

Topic ' 

D* Suomary of Feedback froa 
Field Testing 

£• Sumary Comoents froa Field Staff 

BREAK 

F* Discussion of Phase 4 Activities 
6» Wrapup 



Presenter Tine 

Tony • 9:00 - 10:00 

• 10:00 - 10:45 

Kast 11:00 - 11:30 

Tony 11:30 - 12:00 



94 
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APPENDIX D 

. Questionnaires Ad^Hn^c^»vn d Durinfr .1.,^ . ff ^^^-'m: 
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STODSNT, INSTRDCTION, AND STAFF INFORMATION 



Hov iaportant is infonatloa fors 



I. STDDENT IHPOHMAIION 

A. Background Inforaatlon 

Date of birth 

Place of birth 

Ethnicity/language group 

Date began U.S. residence 

Language used in home 

Fluency in English as of 

Fluency in native language (LI) 
if other than English as of 

Years of education completid as of 

Years of education missed as of 

Family income status (e.g., AFDC, 
participation in National School 
Lunch Program) 



Public or private school 

School grade level 
Sex ■ 

B. Service Data 

Dates entered and exited project 

Other instructional settings (e.g., 
mainstream class. Chapter 1 program, 
special education, migrant program, 
ESL program) 

Dates entered and exited other settings 



Estimating Program 
Effectiveness 
Least Most 
Important Important 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 
3 



1 2 

1 2 
1 2 



4 5 



3 
3 



4 
4 



12 3 4 
12 3 4 



12 3 4 
12 3 4 



5 

5 

5 
5 



1 2 3 4 5 



5 
5 



Improving Program 

Performance 
Least Host 
Important Important 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 



ERIC 



D-2 



S6 



• 




Hov important is infonaation for: 




• 


Estlmatlug Program 
Effectiveness 


Improving Program 
Performance 




C. Hoxt-^Test Data 


Least Most 
Important Important 


Least Most 
Important Important 




Years retained in grades as of 


1 2 3 4 5 


1 2 3 4 5 




Whether or not student dropped out 


r 2 3 4 5 


1 2 3 4 5 




Days absent from project classroom 
as of — and total school days 
possible as of that date 


1 2 3 4 5 


1 2 3 4 5 




Date of special education referral 
or placement 


1 2 3 4 5 


1 2 3 4 5 




Date of placement in gifted and 
talented program 


1 2 3 4 5 


1 2 3 4 5 




Enrollment in postsecondary 
education institution 


1 2 3 4 5 


1 2 3 4 5 




Setting from which entered project 


1 2 3 4 5 


1 9 ^ A S 

•L ^ O H J 




Grade point average ^riar to entry 
in project 


1 2 3 4 5 


.1 2 3 4 5 




Setting to which exited from project 


1 2 3 4 5 


1 2 3 4 5 




Grade point average after exit 


1 2 3 4 5 


1 2 3 4 5 


• 


D. Fomer Project Students 








Standardized achievement test data 


1 2 3 4 5 


1 2 3 4 5 




Classroom performances (e.g*, grade) 


1 2 3 4 5 


1 2 3 4 5 


II • INSTRUCTION 








Academic and non-academic subjects 
taught in the project 


1 2 3 4 5 


1 2 3 4 5 




Method of service provision (e.g., 
tutoring, classroom instruction) 


1 2 3 4 5 


1 2 3 4 5 
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The following variables apply to each class 
or subject matter offered by project: 

Hov laportant Is InforBation for: 

Estitaating Program Improving Program 
Effectiveness Performance 



. Least Most Least Most 

Important Important Important Important 

English and LI materials and how they 

are used 12345 12345 
Percents of instruction conducted in LI 

and in English 12345 12345 

Pattern of classroom language use 
(e.g., concurrent, preview/review, 

or alternating) 12345 12345 

Critical teaching strategies including 
instruction and management procedures 
(e.g., use of active teaching behaviors, 
use of content from students' home 

culture) 12345 12345 

Locations of instruction (e.g., 
mainstream classroom, special 
bilingual classroom, newcomer center, 

resource room) ' 12345 123 4 5 

Student groupings (e.g., LEP students 
are grouped by English proficiency, 

all LEP students are together all day) 12345 12345 
Student/ teacher ratio, student/aide 

ratio per class 12345 12345 

Variability of students within a 
classroom (e.g., age, grade, educa- 
tional attainment, home language, 

language abilities) 12345 12345 

Function of aides in classroom 

(% instruction, % management) 1234 5 12345 

Form, of instructional group (e.g*, 

whole class, tutorial) 12345 12345 

For each subject area taught, minutes 
per week devoted to specific activities 
(e.g., math workbooks, English spelling, 

language lab) 12345 12345 
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III. ST&FP 



&09 inportant is infotnatlon for: 



Academic preparation of each staff 
member, including: 



Estimating Program 
Effectiveness 



Improving Program 
Performance 



Least Most Least Host 

Important Important Important Important 



• level of education, 

• academic preparation in 
field of education, 

• credentials and certificates, and 

e academic preparation in field of 
bilingual/bicultural education 

Teaching experience of each staff 
member, including: 

• years of experience in 
monolingual classrooms, and 

• in bilingual classrooms 

Language abilities of each staff 
member, including languages understood 
and spoken and degree of fluency in: 

• English 

• LI 



1 2 3 4 5 



12 3 4 
12 3 4 



12 3 4 
12 3 4 



12 3 4 
12 3 4 



5 
5 



12 3 4 5 



5 
5 



5 
5 



1 2 3 4 5 



1 
1 



1 
1 



1 
1 



2 
2 



2 
2 



2 
2 



3 
3 



12 3 



3 
3 



3 
3 



4 
4 



A 
4 



4 
4 



5 
5 



5 
5 



5 
5 
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P&OJECI MCK6R0UHD INFOBHAXION 



Coaaualty Characteristics 

lo Percent of c<»aunlty residents- belov federal 
poverty lineo 



o How accessible is this information? 



Not 
Accessible 

12 3 



Easily 
Accessible 
4 5 



o How useful is this data for providing 
background information, about project? 



Not 
Useful 

12 3 



Very 
Useful 
4 5 



2. Unemployaent rate* 



o How accessible is this information? 



o How useful is this data for providing 
•background Information about project? 



Not Easily 
Accessible Accessible 

1 2 3 4 5 

Not Very 
Useful Useful 

1 2 3 4 5 



3. Workforce composition (blue collar, white collar, professional). 

Not Easily 
Accessible Accessible 

o How accessible is this information? 1 2 3 4 5 



o How useful is this data for providing 
background information about project? 



Not 
Useful 

12 3 



Very 
Useful 
4 5 



A. Ethnic aakeup of coaaunity (Asian, Blade, Hispanic, Other Minority, 
Non-HlGpanic Caucasian)* 

Not 



o How accessible is this information? 



Accessible 

12 3 



Easily 
Accessible 
4 5 



o How useful is this data for providing 
background information about project? 



Not 
Useful 

12 3 



Very 
Useful 
4 5 



5. Average length of residence in oraatunity. 



o How accessible is this information? 



o How useful is this data for providing 
background information about project? 



Not Easily 
Accessible Accessible 

1 2 3 4 5 

Not Very 
Useful Useful 

1 2 3 4 5 
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6« Percent o£ single-parent faailie8« 



0 How accessible is this information? 



Not 
Accesdlble 
1 2 



Easily 
Accessible 
4 5 



o How useful is this data for providing 
background information about project? 



Not Very 
Useful Useful 
1 2 3 4 5 



7. Educational level of adult coazumlty residents (no high school^ sosb high school^ 
high school graduate^ seme collegOy college graduate). 

Not Easily 
Accessible Accessible 
0 How accessible is this information? 12 3 4 5 



0 How useful is this data for providing 
background information about project? 



Not Very 
Useful Useful 
1 2 3 4 5 



8. Cocnmnity attitude towalrd bilingual education (generally posit Ive, neutral or 
aixed, generally negative^ don't knov). 

Not Easily 

Accessible Accessible 

0 How accessible is this information? 12 3 4 5 



0 How useful is this data for providing 
background information about project? 



9« Others: 



Not Very 
Useful Useful 
1 2 3 4 5 



District Deatographlcs 
1. Approxioate peak student enrollisent. 

0 How accessible is this information? 



Not Easily 
Accessible Accessible 
1 2 3 4 5 



0 How useful is this data for providing 
background information about project? 



Not Very 
Useful Useful 
1 2 3 4 5 



2« Number of schools in district. 



0 How accessible is this information? 



Not Easily 
Accessible Accessible 
1 2 3 4 5 



o How useful is this data for providing 
background information about project? 



Not Very 
Useful Useful 
1 2 3 4 5 
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3* The school district is characterised as (rural, suburban, urban, inner city). 

Not Easily 

Accessible Accessible 

0 How accesplble Is this Information? 12 3 4 5 

Not Very 

o How useful Is this data for providing Useful Useful 

background Information about project? 12 3 4 5 

A. Ntxnber of classrocsis in bilingual prcgraa in general e 

Not Easily 

Accessible Accessible 

o How accessible Is this Information? 12 3 4 5 

Not Very 

o How useful Is this data for providing Useful Useful 

background Information about project? 12 3 4 5 

5* Percent of students eligible for free or reduced^-price lunch* 

Not Easily 

Accessible Accessible 

o How accessible Is this Information? 12 3 4 5 

Not Very 

o How useful Is this data for providing Useful Useful 

background Information about project? 12 3 4 5 

6* Ethnic makeup of student body (Asian, Black, Hispanic, Other Minority, 
Caucasian) • 

Not Easily 

Accessible Accessible 

o How accessible is this information? 12 3 4 5 

Not Very 

o How useful is this data for providing Useful ' Useful 

background information about project? 12 3 4 5 



7. Others: 



School Characteristics 

1* Student enrollaent* 

Not Easily 
Accessible Accessible 

o How accessible is this information? 12 3 4 5 

Not Very 
o How useful is this data for providing Useful Useful 

background information about project? 12 3 4 5 
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2* Perceot of students eligible for free or reduced-price limches^ 

Not Easily 

Accessible Accessible 

o How accessible is this information? 12 3 4 5 

Not Very 

o How useful is this data for providing Useful Useful 

background information about project? 12 3 4 5 

3* Ethnic saakeup of student body (#slan» Blacky Hispanic, Other Minority, 
Caucasian) • 

Not Easily 

Accessible Accessible 

o How accessible is this information? 12 3 4 5 

Not Very 

o How useful is this data for providing Useful Useful 

background information about project? 12 3 4 5 

4* Number of bilingual teachers (non-Title VII, Title VII)* 

Not Easily 

Accessible Accessible 

o How accessible is this information? 12 3 4 5 

Not Very 

o How useful is this data for providing Useful Useful 

background information about project? 12 3 4 5 

5* Number of bilingual aides (non-Title VII, Title VII). 

Not Easily 

Accessible Accessible 

o How accessible is this information? 12 3 4 5 

/' 

/ Not Very 

o How useful is this data for .providing Useful Useful 

background .information abquc project? * 1 2 3 4 5 

6. Number of bilingual classes (non-Title VII,. Title VII). 

Not Easily ' 

Accessible Accessible 

o How accessible is this information? 12 3 4 5 

Not Very 

o How useful is this data for providing Useful Useful 

background information about project? 12 3 4 5 

7* Others: 
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TITLE VII BILINGUAL SPECIAL EDUCATION 
TEACHER SURVEY 1986-87 



each item. • Circle all answers that apply to 

^SL° ^'Xl'"-*"^ " - When transferring a student to 

a. Fre,„en=y with .hlch a student uses English In speaUng. 

^yonl!'' be at ccunlcatlve level or " 

0. student's reading skills In Spanish are at second grade reading level 

d. Sfdent has been taken out of Spanish reading. 

e. Student has been elven Fnon«u 

given English as a second language instruction. 

f. Student reads orally in English. 
8. All of the above. 

Comments: 



2. 



Which criterion do you think in t-u^ ^ . 

Before «ansfarrlng^r;S:\oX«rEei:JSr" ' " 
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Which of the following procedures would you use in a bilingual classroom? 

a. When a student doec not understand a task in English, I use Spanish 
to clarify the instruction. 

b* When conducting a lesson, I give directions and teach in one language, 
but use the other language when making informal comments to the 
student. 

c. Picture clues, gestures or other non-language techniques are used to 
clarify instruction rather than translating to the other language. 

d. Comments: 



I use the following techniques in a Bilingual Special Education classroom 
for teaching content: 

a. When introducing a new concept to the total class, Spanish is used. 

b. Both Spanish and English are modified according to the levels of 
proficiency of the students in both languages when inl-,roducing a new 
concepts 

c. All of the above. 

d. Other (specify)^ 



When teaching English reading; if a student does not understand a passage 
and asks a question in Spanish, 

a. I answer in Spanish. 

b. I answer in English. 

c. Sometimes I answer in Spanish and other times in English. 

d. I make the student ask again in English. 

e. I use gesture, picture clues, etc. to clarify. 
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When teaching Spanish or English reading, 

a. I use the preview/review method. 

b. I use the concurrent translation method. 

c. I use the phonics method. 

d. I use the whole language approach. 
e« It depends (specify) 



8. 



9. English as a second Ungoaga skills are Ust acquired If: 

b. The teacher uses only English. 

c. The teacher corrects errors in students' speech. 

d. The student is allowed to use Spanish. 

e. By intermediate age. students are interacting verbally in English. 

^^^^^ «e important in promoting 
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APPENDIX F 



Salem-Kei zer Public Sc^iools Dociiment 
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SALEM-KEIZER PUBLIC SCHOOLS 

Bilingual Education Program 
Observation Form 



Teacher 



Observer 



Date 



Time 



Aide 



Other 



Large Group 



Small Groups 



Number of 
Students 












Content 
Area* 












Method ology+ 












Appropriate 
(1 - 5) 












Language ot 
Instruction 












Appropriate 
(1 - 5) 












Appropriateness 
of ESL for 
Students (1-5) 












Appropriateness 
of Language Use 
oy Instructors 
(1 - 5) 












Students' Time 
on Task (1 - 5) 













Ratings: (1 = Low; 5 = High) 
* Content Areas: 



ESL 

Math 

Art 

Language Arts 
Science 



Reading 
Music 

Social Studies 
Physical Coordination 
Perceptual Memory 



+ Methodologies; 

Simplified Teacher Speech 
Modeling and Gestures 
Comprehension Checks 
Academic Language Development 

- Clustering and Webbing 

- Schema Building 
Cooperative Learning 



TPR 

Pre-teaching 

Vocabulary 

••♦development 
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Program Documents from Valley Center. California 
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Program Goals and Objectives 

.1) To assist limited English pro-flclent children to 
achieve English language pro-f I c i ency . 

To develop the child's primary oral language (Spanish). 
1.1 By June of each program year, project'partlcipants that 
score at or. below Level 4 on the BSM in Spanish and who have 
been in the program -for nine months, will advance an average 
o-f one level In Spanish -fluency as Indicated by pre/post 
test scores on the BSM test. 



^''^ f^^^^^ being In the project -for a minimum o-f nine months 
participants that score at or below Level 4 on the BSM in 
English who are receiving English as a Second Language CESL) 
instruction will have on an average advanced one level in 
English -fluency as indicated by pre/post test scores on th*» 
Billpgual Syntax Measure. 

2) To raise the academic achievement level o-f limited 
English pro-f icient children. 

2.1 K-8 project participants who receive reading 
instruction in Spanish and who have been In the program a 
minimum o-f nine months will gain on an average 1.2 months 
growth per month o-f instruction as measured by scores on the 
Reading and Math subtests o-f the Comprehensive Tests o-f 
Basic Skills <CTBS-Espanol ) . 

2.2 LEP students in grades 2-8, who have been in the 
program since kinder or a minimum o-f two years will have 
gained on an average 1.2 months per month o-f instruction in 
Reading, Language and Mathematics as measured by the 
Comprehensive Tests o-f Basic Skills In English. 



3) To develop Spanish as a Second Language -for 
children whose primary language Is English. 

3.1 On an average, participants who receive Spanish as a 
second language <SSL) Instruction will at the end c-f the 
-first year pass 5X, the second year 15%, the third year 25JC, 
the -fourth year 50J<, the -fl-fth year 70X, the sixth year 85i<, 
and the seventh year 100% o-f the items on the district made 
test that measures receptive and expressive skills in the 
second language. 
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t:L/^.?it P*'^*'^'.'*-'^*^ who haye been in the program four 
years wi 11 score *n iwer^ge of a Level 2 on the BSM Spanish 
Each .year these participants will continue to orow an 

rZ2r. revii'^ra.^" ^^^"^^'^ -^^^ '^-^ 

English prSfriiinTsUtu;?'*'^ proficient students to fluent 

t^Lf^.l^t ^^^'^^ project" limited EnoUsh 

prof CI en t students on an average 80% will be reclassified 

D?Jn?j;*.^"^','^*' proficient status as determined by he 
District's reclassification criteria. 

fK.m.^^ develop students positive attitudes about 
<i themselves and others. 

^fhln ^t^d^*^*^ in their primary languaoe and 

then, transitioning them into English the positive self 
esteem of the students will improve as noted on the 
self -concept survey. 

5.2 Project participants will Interact social ly. 

6) To develop multicultural appreciation of the 
o?hers? ^^'^'^^^^ °^ limited English proficient children and 

.^.'.l^•^^ff'^®^^,'^:^^ biUngual program will ensure that 
multjcultural/multiethnic curriculum Is integrated 
throughout the curriculum. Student quest ionnal res wl 1 1 be 
tiirgoT ^"^ multicultural issues to assess completion of 

7) To continue to develop and re-fine each project 
component area. k - w 

• 

HoLfL*!;^ ^''''^ project year the project staff will 

develop or refine a minimum of one component of the fIvA 
inuifuL.'nr^ (instructional program, parent and com,r.unity 
involvement, staff development, evaluation of program, 
curr I culum materials) . yi 

8.) To continue the training for the bMlnoual staff 
working with the LEP students. ^■•nguai staff 

8.1 By the end of each June project staff working with LEP 
students will attend a minimum of one inservlce related to 
bilingual methods and techniques. 
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TYPES OF STUDENTS 

.he following is an example of five types of students which we service 
in the Bilingual Program. Read through the following list and if you 
disagree with it please make changes which you feel are appropriate. 

1. - ENGLISH ONLY students who have been in the program since kinder 
and who's Spanish skills are well developed. 

2. - ENGLISH ONLY students who entered the program after kinder and who 
have poor second la ^uage skills. 

3. - LIMITED ENGLISH PROFICIENT students who have been in the program 
since kinder and have average or above averag^^ language and math 
skills (LI). 

4. - LIMITED ENGLISH PROFICIENT students who have been in the program 
since kinder but, who have poor language and/or math skills CLl). 

5. - NON ENGLISH PROFICIENT students who 'enter the program after "kinder 
and have no English skills. 



PROBLEMS IN PROGRAM IMPLEMENTATION 

1. - Limited English profici-ent students who enter mid-year and have 
little or no school experience. 

2. - Great variability in the second language skills of all students. 

3. - Great variability in (LI) skills and levels of maturation in 
limited English proficient students. 

4. - English only students who are put into the program without parent 
request or knowledge to even out numbers in other classrooms. 

5. - Limited visuals and concrete objects for teaching subject content 
for mixed language groups. 

6. — Teachers should be able to reassign students to ED zl. r»rooms if 

they feel they can't handle a second language. 

'^••^ Teachers giving tests should be the daily classroom teachers which 
the students know and feel comfortable with. 

8. - Not all parent notices which go home are transalated into Spanish^ 

9. - Need a secretary or part time office clerk who speaks Spanish. 

10. - Need inser vices for teachers who have students in ESL stage III. 

11. - Need Special Education and other special services for Spanish 
speaking students. - 




OBSERVATION 



«K4. •^'^f PUf^Pose of this observation is to confirm tlie data 
obtained -from tlie bilingual teachers as critical 
instructional features. The information oathered by the 
observation is to confirm this information. 

Instructions for the observer- 

A classroom win be observed once a week for two weeks, 
.he observations will be conducted for a period of 15 
minutes three times a day: at the beginning of the day, 
mid-morning, and late afternoon. Each observer will decide 
the exact time of his/her observation after having discussed 

5k! *k!^^''w°'^,^*"^^*?"'* ^^""^ individual teacher. One of 

the three daily fifteen minute observat i o.is must include a 
five minute observation of a lunch or recess period. The 
observer will use the time chart in the following page to 
schedule his/her observations. 

The observer is required to answer all questions in the 
instrument during th^ early morning observation. During the 
micJ-morning and late afternoon observation only the 
questions without asterisks * are to be answered. In the 
last question of the observation instrument the observer is 
encouraged to add any information which s/he fe^^ls might 
supplement previous observations. 
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Critical features of the Bilingual Projxraai a the Valley 
Center Onion School District . 



Global Critical 
Features 



Specific Critical 
Features 



Data Soun 



Use df student's 
first language CLl) 
for core subjects 



tdtMathi^ reading and language are 

taught in the student's 

home language until 

they develop fluency 

In their second language. 

Students are encouraged to aalntaln 

their first language. 
-ktht Bilingual Syntas Keasure Test £ind 

a hoae language survey are used to 

deteraine vhat language a student vill 

be taught in. . 



Observation 



Ke cords 



Jse of student's 
second language (L2) 
for aon core subjects 



*Art, science^ social 

studies, fiusle (math) are 
taught In the students 
second language* 



Observation 



ilxed group 
instruction 

>tuddfxts are encouraged 
;o help one another (role 
4odel'iag for second language 
icqyisitioa) 



:ftClassrooas are foraed 

50% Spanish speakers 

50% English speakers 
^Students are taught alternating weeks 

English/Spanish giving equal status to 

both langua^vts. 



Observation 



Multicultural teaching 
approach. The culture of 
•he three aajor ethi^lc groups 
^presented Is given erqual 
aportance. Other cultures are 
ncluded In the curriculum. 



:ftCurrlculuA aaterlals 
contain inforsatlon 
about students' hoae 
culture. 



Lesson plans 
Revleii aaterlals 



caching second language 
:qulsltlon through the 
Jatural Approach'' • 



aching through active 
.rtlcipation. 

ERiC 



*Use of the Total Physical Response 
teaching technique. Teacher uses 
voice entonatlon, gestures^ 
aanipulatlves, and visuals to teach. 
Students leara poe&s# songs* draaas^ 
and write stories using their oun 
language. 

^^Participation of EO students 

In prograa; they are encouraged to 

learn a second language. * 

tfcStudents rotate around learning - 
centers using hands on activities. 
•Cooperative Learning" techniques are 
encouraged* ^ . ^ 
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Observation 



Observation 
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NAME OF OBSERVER. 

DAY- 



.DATE 



STARTING TIME ^ ending TIME. 

WHICH OBSERVATION IS IT? 



_Early morning 

2n <i _M i d-mor n i n 9 

^^'=^ _-___Late a-fternoon 

NAME OF TEACHER BEING OBSE.WED__ 

* 1.) Type o-f classroom; 

Se 1 -f-con ta i ned 

^Team -teach ing 



* 2. ) ■ Grade level 



3.) What subject is being taught?. 



Number times te^^rh^r sDe^k<. cf. , dents in gn.ni.K 



Number o-f times te^rh^r SDe^l^^ ^ tudent<. in Pn^ i;.K 



Number of times stndp n t ansu.pr<. teacher in Rp:.A!ch 



Er|c """^ 116 



Number o-f fin.es i^furt^nf .nsu.Pr.. f...K . p ;„ p„^, 
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-VALIDATION OF OBJECTIVES- 
VALLEY CENTER &IHNGUAL PROGRAM 



READING AND LANGUAGE OBJECTIVES \ GRADE 



i Is the objective appropriate for your grade level? 



yes. 



no 



why 



Does the objective include important developmental 
skills the student^must. master for next year. 



yesj no. 



: Does the objective cover skills that you. teach. yes no 

o (If you answered ho on this question skip questio.-- • 

vo D and E for this objective) 



) Check the approximate number of sUiUs taught to meet " PI I — \ \ \ 

this objective ' — I I | 1 1 

1-2 3-6 7-12 

Check the approximate numb^^r of tirae soent teachlnn fh^ 4.1 . ^ 

obj.ective during the year ^ teaching the p-, throughout f~\ 1 to 2 f-^ 1 to 

« — I the year I — I months V.... I weeks 



List other skills you teach (in 
and which wore not n^^ntioned in 
question is to be answered only 



reading and language) 
the obJt?ctives. (This 
once) 
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C.T.B.S. TEST OBJECTIVES 

pBJECnVOS 

nivel 


1 

i 

A 


1 

Bl 


C 


i5 
\ 

1 


2 


f 

3 




Reconocimiento del sotiido 

J . El estudiante idcntif icara correctamente la 
t^AwujLQ ii.Ti(ict oicijjTientie. 




0 








1 




2. El estudiante identificar^ correctarnente la 
palabra que ^signif ica algo sohre una palabra 
o una oracion dada oralmente. 
















Cjooprensioi de lectura 

3. El estudiante ident if icara correctarnente el 

dibujo ^ue ensena el significado de una oracion' 




^^ 










4. El estudiante identificar^ correctarnente la 
palabra que significa algo sobre un dibujo que 
vee. 




0 










C Ex estudiante identificara correctarnente la palabra 
tjue iv^joz earpiece la oracion escrita* 












... 

{■\ 

* I 


6. El estudiante identificara'' correctarnente la 
pdlabra que significa algo sohre la palabra 
o frase cue esta submuarta 










7. El estudiante contestara^ oorrec^tamente pre- 
guntas cono: qiiiai, qah, donde, .y cuafido, 
sobre rasaies oue lee* 














120 


• 













C.T.B.S. TEST OBJECTIVES 



- — ^xiivei" 


A 


B 


C 


1 


2 


3 


Cbmprension de lectura 

8* El estudiante podra ancilisar sentimientos 
o motivos de los car^t^res en un pasaje 
que lee. 






)^ 

O 


y 
C 


o 


o 


9. El estudiarite identificara la idea prin- 
cipal, el intencion del a.utor, o el hunor 
y el tono expresado en tin pasaje que lee. 






0 


17 

o 


i: ) 




10. El estudiante sacar^ concltisiones del pasaje 
que lee. 




■ 


■)i 
0 


o 




^- 


uarputacion de matematicas 

11; El estudiante sumara'^nuneros en^reros. 






o 


/• 


1 


t 

• 


12. El estudiante sumar^ decimales o nunaros 
quebrados. 












( '• 


13. El estudiante restart nirnaros enteros. 






0 ( 


o 




14. El estudiante restara deciuales o 
nirnaros quebrados. . 














15. El* estudiante multiplicar/ nirveros enteros. 








n 


■ ^ 




16. El estudiante inultiplicara decimales o j 
nitfneros quebrados. | 










n 


c 


17. El estudiante dividara niJneros enteros. \ 








o 




18. El estudiante dividara deanales o 
nuneros quebrados. 
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C.T.B.S. TEST OBJECTIVES 



Conceptos y i^licaciones de matonafcicas 

19. El estudiante denonstrara^ que entiende 
la nvunsracion. 



nivel 



B 



1 



0 



20* El estudiante denpnstrara que entiende 



o 



21. El estudiante demOTstrara que entiende 



22. El esj:6aiante dononstrara que eitiende la 
teoria de numeros. 

23. X El estix3iante deronstrara que enti^de 

me didasy dineip^ V tienpo . 



24* El estudiante denonstraraT que- entiende 
gecmetria. 



o 



.0 



I.) 



I ) 



« » 



' J 



o 



0 



o 
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C.T-BaS. TEST OBJECTIVES - VALIDATION 
First grade READING 



RECONDCIMIENTO DE SONIDO 

1.- El estudiante identificara correctamente la palabra dada 
oralmente. #l*6b of VC OBJECTIVES 

COMPRENSION DE LECTURA 

3. - El estudiante identificara correctamente el dibujo que 
ensena el significado de una oracion escrita. #1.6c of VC 
OBJECTIVES 

4. *- El estudiante identificara correctamente la palabra que 
significa algo sobre un dibujo que ve. #1.6d of VC 
OBJECTIVES 

The greatest concern among first grade teachers involving 
these three objectives was how far into the reading sequence 
the objective measured. 

Spanish reading in Valley Center is taught in a phonetic 
global manner with a continuous emphasis in oral language 
development. The child is first taught the vowel sounds and 
their combinations. The next step is to introduce consonant 
sounds one at a time and from the very beginning whole 
sentences are stressed. These sentences contain only the 
vowels and the consonant (s> which have been introduced. A 
specific sequence is followed when introducing consonants 
the first seven letters being M,S,P,T,L,Dpand N. 
The above objectives do not state if only certain consonants 
are used or i f the child is expected to know the entire 
alphabet. If the child is expected to read words containing 
all the letters of the alphabet only about 50% of the Valley 
Center first graders (who read in Spanish) would attain 
these objectives. 
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C.ToB.S. TEST OBJECTIVES - VALIDATION 
Second grade - READING 

RECONOCIMIENTO DEL SDNIDO 

EI estudiante identificara correctamente la palabra que 
significa algo sobre una palabra o una oracion dada 
oralmente. #2. 3b VC OBJECTIVES 

COMPRENSION DE LECTURA 

5.- El estudiante identificara correctamente la palabra que ' 
mejor complete la oracion escrita. #2.4 VC OBJECTIVES 

8. - El estudiante podra analisar sentimientos o motivos de 
los car ac teres en un pasaje que lee. #2.5 VC OBJECTIVES 

9. - El estudiante identificara la idea principal, la 
intencion del autor, o el humor y el tono expresado en un 
pasaje que lee. #2.6 VC OBJECTIVES 

10. - El estudiante sacara conclusiones del pasaje que lee. 
#2.8 VC OBJECTIVES 



Second grade teachers agreed that all of the above reading 
objectives were appropriate for their grade level. They, 
agreed that the objectives included important developmental 
skills the students needed to master for the following year. 
They all taught a large number of skills throughout the year 
to meet these objectives. 
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C.T.BoS. TEST OBJECTIVES - VALIDATION 
Third grade - READING 

COMPRENSION DE LECTURA 

B-*" El estudiante identificara correctamente la palabra que 
significa algo sobre la palabra o frase que esta subrayada. 
#3. 5a ?c b VC OBJECTIVES 

7.- El estudiante contestara correctamente preguntas como: 
quienr que, donde, y cuando, sobre pasajes que lee. #3.6 VC 
OBJECTIVES 

El estudiante podra analisar sentimientos o motivos de 
los caracteres en un pasaje que lee. #3.12 VC OBJECTIVES 

9. - El estudiante identificara la idea principal, la 
intencion del autor, o el humor y el tono expresado en un 
pasaje que lee. #3.7 VC OBJECTIVES 

10. - El estudiante sacara conclusiones del pasaje que lee. 
#3.9 VC OBJECTIVES 



Third grade teachers agreed that all of the above reading 
objectives were appropriate for their grade level. They 
agreed that the objectives included important developmental 
skills the students needed to master for the following year. 
They all taught a large number of skills throughout the year 
to meet these objectives. 



ERIC 



-f rt rr 



EVALUATION QUESTIONS 



1. ) Are students who have been in the billngu*! proqram 
iour years doing their core subject in their second ' 
language? 

2. ) .During unstructured play do English and Spanish 
speakers interact socially? 

'3.> Does the teachers' attitude towards Bilinoual Education 
affect the students perception about learning I second 
language? • 

4. ) Is the success of the program due to the methods 
teachers use or more to the fact that we are good teachers 
and care about the students? 

5. ) Do Spanish component teachers use more English than 
they should? 

Is the success in ESL due to good ESL instruction or 
exposure to English in the environment <on playqround, bus, 
iv, etc.) ~ ' 

7. ) Is the program meeting the needs of non Enlgish 
speakers who enter the program after second grade? 

8. ) Are students ih'bilingual classrooms truely more 
cul tural ly aware? 

f.) How do the students in the prooram feel about b^inq 
inyolved in the program? 

10-.) Have the changes that have been made in the program's 
component areas (parental involvement, staff trainino, 
evaluation, instruction, and curriculum and mater iali) been 
useful? 

11. ) Are students who have par t i c i.pated in the project 
graduating from high school? 

12. ) Are students who are participating in the team teachinq 
approach learning Spanish faster than those in the s^lf 
contained? 

13. ) Are fewer students being retained? 

■ V ■ 
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AFFENDIZ H 

San Jose Unified School District Doctmient 
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CALIBRE/VOICES CuVS^lZ^M fdtUA 



1. Amount of time with computer assisted Instruction:, 

2. Language used byl Instructional aide: 



SAMPLE FORM 



Instructions: 



I. Background 

A. Name of aide'/tfeifehei':. 

B. School: 

C-. 
0. 
E. 
,F. 



Subject area: 

No. of students observed: 



T^pe of bilingual methodology eraploywU 

r\ J.- - Type of learning environment: g3g^^Mm 

^U. Observation Data (teacher/an dej * — 

7? A. Language \is^(-esHtiULte) — 

L, % Qjidt I 



Small groui^ Individual^ 



2. 



•1 



% 



3. other (please^specify) 
Time on learning task 
/ 4* 



^2. 



2 4, 

3 4-. 



with teacher: 
with aide: 



minutes 



BinutP'J 



with computer assisted program: 
C. Student Participation - 



jninutes 
it 



1. 
3. 



■ number of student rcsponaes; 
fypn nf r ttidont rocp o n sa^ : , 



D. 



are students Involved In relevant follow-up activities? Yes_ 
Integration of Cultural 

1. use of relevant gvampiQQ» . 



No 



o 
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E. LearnijfgiEnvironmen 

1. /s the person^s lafeiguage (L^^ 

2. /Are ci»^^uj»1iy rel 
Other (please spec 

in. Observer Information 

1. Name; 

2. Date: 

3. 




Duration of Observation :_ 
Comments : 



Observer's Signature: 



^>p^^a5te'subEii>&^th\s_cflap-reted 
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COMMUNITY SCHOOL DISTRICT TEN 
BOARD OF EDUCATION OF THE OTY Of NEW VEXttC 
rj. M*9M1 HftUyiAN AVENUE 
•nO»OC»RY.104« 



FnEDOOCCCM CAMORALEWSl 



SugRCated Criteri a for English As A Second Language ClasTooa Observaraona 
A« The t eel on 

1. It Che language uiad appropiate for student*! level of 
profic iency? _ _ 

2. Are worda and forma wad ii context? (In queationar aenteariiT'ind"' — 
situations) • • ^ • 

3. Are tha aim of the various eajor aectiona of the lesson (reviev of 
the previous vork» boardwork, new presentation) clear? 

4. Are tha various »ajor sections of the leason aotivated? 

5. Is the presentation of tha new vork well organized? 
■ 6. Is the prasantatlon of the nav work clear? 

7. Are thare checks aade on the coaprehenslon of the new presenUtlon? 

}l devoted to the new presentation? (e.g. 

lS-20 Mln*) • 

9. Is a najor part of tha lesson devoted to oral activities? 

H^Jw" •^T*'^?^ •ctivitles? (dialogue, question and antvar. singing, 
drills, audio-visual, etc.) 

11. Does the teacher use devices for heightening interest in the lesson 
(personUijetl^n, situational approach, visual supports, etc.)? 
of the St^e^to?*** appropriate to the activity and to the leyel 

13. Are cultural eleaents Introduced as part of the lesson? 

14. Does the lesson have an over-all structure? . (,.g. vara up, review, 
nev aatarlai, eppllcatlon* suffwiry, asnljirMnt, etc.) 

J- Quesdonlp g and Student Partieip agion 

1. Are uost of the students involved in the lesson? 

2. Are quegtloM used for a variety of purposes (eiicitlng, problue of 
coaprehcnsloa, ticn.!orcing, personalizing)? ^V-^ 

3. Are questions clear t , ' -r/ 

4. Are questions individualized? -v * 
vol^SS?"* "udents, is there a.balance between volunteers and non- 

6. What is the quality of Response?? 

Routines ^ ... 

•"1. Are directions dear? 

2. Is the time of the period used econoalckuy and eppropr lately? 

3. Are there treasitions to new activities? " 

4. •* — ^. . . ■ .- 

5. 

ciaterlals? 



_ UBW •CtiVjkCiSSf 

Does the teacher have an effective Banner of starting thi lesson? 
SS^rSLr ^""^ «fi.ctive manner of distributing and collecting 



D« Boardwork and Use of A-tda 

^' ^or^acIL^nf^'' utilization of the boards? (aaslgnBent, new vocabulary, 
sSltS ^UvUl. dictation, drills, fUl-lns, completiiis, 

2. Is aost of the wriclng on the boards done by studeuts? 
J. Does the tcacuar use visual supports effectively? 

^. le the >llustretive aaterlal appropriate and Integrated with -the leison? 
E. The Teacher i • . 

1. Is the teacher prepared for the lesson? 

tJLlh^^l techniques of teaching t\s.t.' 

r"n&Snt)? • "terial, review, boardwork, driUs, 

3. Does :hc ccach^or have rapport with thu fitiulents? 



U the teacher .daptlng the textbook to hi./her clasiSeedsT 

P"?"""! qualities necessary Uf effective teaching 
(vitallcy, concern for all students, eagerness to praise)? 
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RatinR Scfcie for Second tanguage CXtfltrooc Evaluation 



The lagfloa 

Extenc CO vblcb che^ sacond i Un^ 
^oaga la uaad in the claaarooa 

Func clonal 'use of cbe laniuaga 

Koclvaclon ' 

CXarlcy of preatntaclon , 

Evidence of cosprehenalos of 
cbe preaencacion 

Scruccure of tbe preaencacion - 

Eliciclng of inforitaclon 

Dae of vorda in concexc 

Adequacy of drill 

Cxcenc of oral acclviclea 

Over-all acruccure of che loaaon 

Appropriatenosa of acclviclea 

Variecy of acclviclea 

Pace 

' Ihraaaclc QuaXlcj of the-leaeon 

\Sponcaneicy 

Culcural enricfasenc 

Pcrs:?nal.'.i?cion 

Evidence of co-curricular 
acclviclea 

Use of anterlala ocher chan conven- 
Cionel Inacruccional naceriala 
<nowopapera» eagazlnea^ adverciae- 
aenCB, poaters, ecc.) 

Outcoaea 

'.Questioning 

DiacribuCion of quesclona ' 

Individualizacion of quesrionlng 

Claricy of quesclona 

Adequacy of quescioning 

Proper uee of queanionlng (elicic- 
lng, probing, reenforclng, 
peraonalizing) 



Superior 



Good 



Fair 



Unaacla- 
faccorv 
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Ttiagher ' 
?re?arttloa for th* leason 
Ctiftral prtparitlon 
Coioaod of tttchalques 
Llnguiitic skills . 
Rspporc with students 
Vitality , 

Concsra for indlvldusls 
lodepezidencft froa the textbook 
Textbook adaptation 

Students 

Participation - 

Quality of responses 

Scudent«^entered approach 

Individ uallsB t ion 

Icltictlve for tdf-.llractlon and 
cooperation 

Develop&ent of good attitudes 
Routines 



Superior 



Good 



Fair 



Unsatli 

fasserv 



Clarity of directions 

Effictlency in distribution of 
w.tarUls 

Transitions to new activities 
Econonlcar use of tlaie 
Start of lesson 



Boardwork 

Utilisation of boards 
(aasignsent^ new vocabulary » 
correction of aaaigned oaterlal^ 
dictation, drills, fill-ins, » 
coapletions, suttsaries) 

Correction of boardwork 

Student participation In boardwork 

Orderliness 



Use of Aids 

Appropriate use of A-V aids 

Use of teacher or student aade 
materials 

Use of illuscrative oaterials 



.-: •■■'■.•■•\:-»i(S',*J:' 



; ■•■'til:'- 
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Juno 15, t9B7 - Socond Revision 

PROGRAM OBJECTIVES FOR TRnNSTTT QNflL BILTNGUflL EDUCATION PROJECT - 

SRflDES K-3 

LONG RANGE OBJECTIVES (THREE YEAR) 
Student puteome!^; 

I. By tho ond of tho project period, 85Z of the target population 
(these students receiving instruction for tuo or nore years) uill 
demonstrate highwr-order thinking skills obility as neaoured by 
correctly ansuering 75Z of the questions on a program developed and 
program related criterion referenced test, 

\. 

Z. By the end of the project period, 85Z of the target population 
(those students receiving instruction for tuo or more years) Mill 
'domonstrate improvide proficiency in speaking and understanding tho 
second .language (English) by obtaining scores above the Z0th 
^ percentile on the Language Assessment Battery (LAB), 

3« By the end of the project period, the target population 
(those students receiving instruction for tuo or more years * in the 
Thinking Skills Program) uill demonstrate a mean Relative Grouth Index 
(RGI) of over 100 percent in reading in the native language (Spanish) 
as measured by the Comprehensive Test of Easic . Skills, Reading 
Section, Spanish Version, between pretest and positest scores. 

4. By the end of the project period, the target population 
(those students receiving instruction for tuo or more yeiirs in the 
Thinking Skills Program ) uill demonstrate a mean Relalcive Grouth 
Index (RGI) of over 100 percent i;i mathematics in the native language 
(Spanish) as measured by the Comprehdnsive Test of Basic Skills, Hath 
Section, Spanish Version, betueen pretest and posttest scores. 

5. By the end of the project period, the target population (those 
students receiving instruction for tuo or more years in Thinking 
Skills) uill demonstrate a mean Relative Grouth Index (RGI) of over 
100 percent in ^mathematics in the second language (English) as 
measured by the Metropolitan Achievement Test, Hath Section, betueen 
pretest and posttest scores. 
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G. By the end of the project period, the target population (those 
students receiving instruction for tuo cr more years in Thinking 
Skills) uill demonstrate a mean Relative Grouth Index (RGI) of over 
100 percent in reading in the second language (English) as measured by 
Q the Degrees of Reading Pouer,. Reading Test, betueen pretest and 
posttest scores. 
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Praceag nh<ft^;||Yftn- 

J I By *hQ ond Of the ppojoct poplod, 8» popceni of those ataff 
Zlt/^ *r"!"*"° d8«on«trate the ability to offoctlvcJy ulo 

iy a"'«i;n°' *»'i"''i"0 « «v doners 

fL ''i"°.°^^-®°''hlQhoponc)flvopolnt8calStsnt In JSe 

foP„ of tMo chockli.ta of tochnlquB« and bohavloP, (ono for tn^chopa 

Jip a ^inS"'";*'"""' O^ly aidea rocolvlnfl tpalnlnfl 

yoapa in the ppogpan ulll ba Includad. 

P-nM^ln!^ P'"'^'"'' ^® PO'-COnt of thOBO Btaff 

pocBlvlng tralnlno ulll rcpopt on tholp ability to effectively use 

be ni?S«pL °^ innovation. The data ulll 

he QathBPed thpouQh the use of teachep and aide Intopvleus. Only 

teachl5»Q for tuo years In the ppoQpan ulll be Included. 

Parental TnvQ^vl^ff<^^^^; 

1. By the end of the ppoject pepiod, 70 pepcent of taros* 
population papenta ulll demonstrate knouleda; «nd SnderStanSlng oTtho 
llxoY^H. ''"V" ^'^"-tion by correctly ZrZrTnl 

.""'^ °^ project period. 70 percent of taraot 
funcuJr«p'""'''1? have attended one or Sore school -re?^?ed 

aSucauSn. ° connected ulth this pPOQpaH op their students' 

• ' nnn„i°^H*^\r"'^ °l » ^''^a''''" transitional 

nauBl education fpon klndaroarten through third orade unlna 
.xnklno skills ulll have been fully established as evidenced 
docuNentatlon recorded In project loQs. ^ 

r.^.n^' *he ond' Of the project period, a nanual of lessona. 



ERIC 



j-3 



OBSERVATION CHECKLIST 



BACKGROUND INFORMATION 
Observer 



Person (s) Observed 
(1) Name 



Position 
(2) Name 



Position 



— _ Grade _____ class 

Period . 

Subject Matter Taught 



Number of Students on Register . Present 

Length of Observation 



ErIc i37 



CLASSROOM OBSERVATION CHECKLIST 



TEACHER/AIDE 

Thinking Skills Strategies 

1. Does teacher pose open-ended questions? 

2. Does teacher use probing techniques to elicit 
additional information? 

3. Does teacher allow sufficient time for student to 
ponder question? 

4. Does teacher respond to answers in a postive manner? 

5. Does teacher permit and encourage questions from students? 

6. Does the teacher use higher level thinking skills- 
teaching techniques (classifying, analysis, comparison, 
infering, evaluation) • when teaching language arts, science, 
social studies, and math (Based on class observations 
and/or students performance folders)? 
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Methodolocfv 

!• Do lesson plans reflect lesson taught? 

2. Does the lesson have a clear aim? 

3. Does teacher adjust for student ability? 
4* Do slower students receive extra help? 

5. Are charts^ materials and visuals comprehensible? 
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Management 

1. Do students complete work during assigned periods? 

2. Are assignments checked? 

3. Are assignments differentiated by student ability? 

4. Do students have personal achievement records? 

5. Do students share ideas and products of their work? 

6. Is student discipline good? 
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Environment 

1. Do teachers and students have sufficient space to work? 

2. Is there provision for students to work in small groups? 

3. Are there centers for students to work alone or in pairs? 

4. Does the classroom have educationally stimulating decorations? 

5. Are the learning materials w^ll organized and easily accesible? 

6. Does the classroom have visual cultural representation? 

7. Is student work displayed and is it current? 

8. Is there a class library? 

9. .In general, is the classroom environment comfortable and 
conducive to stimulate creativity? 

10. Are there facilities for students to experiment:, and make 
discoveries? 

11. Does visitors log reflect parental involvement? 
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Interaction 

1. Does the teacher demonstrate respect for students? 

2. Do the students demonstrate trust in the teacher? 
Sannerf ""^^ '^^^^^^y and courteous 
and'^aldl.''^^^-^^^" °' collaboration between teacher 
In Sl^chooir" °' cooperation with other staff members 

6. Do students practice peer tutoring? 
rn\^^tltlTil°„l^ '° 
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